Uploading and downloading files to and from a HPC

Question

I am using Compute Canada's HPC cluster and want to download files from the cluster to a local directory. I referred to Uploading and downloading files to and from a HPC for instructions and entered the following command on my Windows Command Prompt (not the HPC cluster terminal):

scp [email protected]:/scratch/file.txt /path/to/local/directory

This requires me to login every time I want to download files.

Since I am working with a large number of files from several directories, I want to avoid having to login every time, since it is very time-consuming. Also, the downloads would be peformed in series (not parallel) if I were to enter them one after the other which also takes a long time. Can anyone explain how you can download files from several directories in an HPC cluster to local directories in parallel without having to login every time?

Answer 1

A few solutions are possible:

1. The best one: set-up a password-less ssh connection. Depending on the policy set in place by the admins of the HPC service, this can be achieved in several different ways, some much simpler than others. Check the restrictions with them and, policy allowing,try following the instructions here.
2. Workaround 1: create an archive. Just collect all the files you want to retrieve somewhere in a to_retrieve directory on the server and create an archive file with for example

$ tar cvfz my_files.tgz ./to_retrieve

or

$ zip -r my_files to_retriev

and then just scp this single archive file that you'll extract once on your Windows machine.

3. Workaround 2: copy several files at once. As scp supports all kinds of unix wildcards and recursive option, you can just take advantage of that instead of copying files one by one. For example, assuming you collected all the files you want to retrieve in a to_retrieve directory as for the first workaround, you can copy them all at once without archiving like this:

$ scp -r [email protected]:/path/to/to_retrieve /path/to/local/directory

NB: Both workarounds will request you to input your password, but that'll be just once for all the copies.

Just up to you to see what works best for you.