Data transfer to and from Windows

Owned by Robert Schulz
23/08/2022
8 min read

You are welcome to use any client that you wish for transferring data to and from a local Windows workstation. There are apps for Windows which allow you to transfer data on the command line or with graphical user interfaces.

Here, we provide instructions for a selection of commonly used apps in addition to the ones mentioned in Login to ALICE or SHARK from Windows

Check out the best practices before starting to move data: Best Practices | Best Practices Data Transfer

PowerShell

If you completed the section on connecting to ALICE or SHARK with PowerShell (Login to ALICE or SHARK from Windows | PowerShell), you are already good to go. The OpenSSH client in PowerShell supports scp and sftp much like on Linux and MacOS. So, you can have a look at Data transfer to and from Linux and Mac OS for the basic instructions on how to use both commands.

RSYNC is not available for Windows and therefore PowerShell. Natively, you can only use scp and sftp.

Windows Subsystem for Linux

Because WSL provides a Linux environment, it offers you not just scp and sftp, but also rsync. You can follow section Data transfer to and from Linux and Mac OS for the basic instructions on how to use all three commands.

MobaXTerm

MobaXTerm comes with an integrated file browser to the left of the command line window:

ALICE

SHARK

When you are connected to the cluster, you can use this window to browse to the directory on the cluster that you want to download or upload data to.

For uploading files, you can either drag-and-drop them from your local workstation to the remote directory in MobaXTerm or you can use the upload icon in the menu which will open a pop-up window allowing you to select one or multiple files

For downloading files, you can also use drag-and-drop or click on the directory in the file browser and then on the download icon.

See the documentation for MobaXterm for additional information.

PuTTY

With PuTTY you can only get command line interface on the server that you connect to, e.g., on ALICE or SHARK. In most cases, this limits how you can transfer files to and from your local Windows workstation. This is because you cannot reach your local workstation from within the cluster.

Therefore, PuTTY alone cannot be used to transfer data from your local workstation to the cluster.

X2Go

X2Go gives you a remote desktop on the cluster. It is not possible to drag-and-drop files from your local workstation as is for example with MobaXTerm. However, you can tell X2Go to mount a directory on your local workstation in your X2Go environment on the cluster. This way you can copy data between your local workstation and the cluster.

Instructions on how to set this up in X2Go can be found here: Getting a remote desktop on ALICE or SHARK with X2Go | Mounting a local folder to the X2Go session

Do not use the local directory mounted with X2Go in any compute job. Always copy the data to the cluster first.

Other sftp clients

WinSCP

In addition to copying files, WinSCP also allows you to synchronise data between your local workstation and the cluster. Here, we provide instructions on how to setup WinSCP for ALICE and SHARK

 

ALICE

SHARK

 

ALICE

SHARK

When you start WinSCP for the first time, you need to setup a site/session.

First, we need to setup the tunnel through the ALICE ssh gateway. To this purpose, click on “Advanced”.

 

When you start WinSCP for the first time, you need to setup a site/session.

If you want to set up a connection from outside the LUMC, click on “Advanced” and proceed to the next row.

If you want to set up a connection for working from within the LUMC network, skip the next row and go directly to the row thereafter.

 

In the Advanced settings, select “Connection” and

  • Enable “Sending of null SSH packets” in section “Keepalives”

  • Set “Seconds between keepalives” to 60

In the Advanced settings, select “Connection” and

  • Enable “Sending of null SSH packets” in section “Keepalives”

  • Set “Seconds between keepalives” to 60

In the Advanced settings, select “Tunnel” in the category “Connection”:

  • Tick “Connect through SSH tunnel”

  • Set “Host name” to: ssh-gw.alice.universiteitleiden.nl

  • Set “User name:” to your ALICE user name

  • Finish the settings by clicking “OK”

In the Advanced settings, select “Tunnel” in the category “Connection”:

  • Tick “Connect through SSH tunnel”

  • Set “Host name” to: res-ssh-alg01.researchlumc.nl

  • Set “User name:” to your SHARK user name

  • Finish the settings by clicking “OK”

This setting is only necessary when you are working from outside the LUMC network.

Back on the login screen:

  • Set “Host name” to login1.alice.universiteitleiden.nl or login2.alice.universiteitleiden.nl depending on which login node you want to connect to

  • Set “User name” to your ALICE user name

  • Click on “Save”

    • in the pop-up window, specify a site name for this connection, e.g., “alice1” or “alice2” and then click “OK” to confirm

 

Back on the login screen:

  • Set “Host name” to res-hpc-lo02.researchlumc.nl or res-hpc-lo04.researchlumc.nl depending on which login node you want to connect to

  • Set “User name” to your SHARK user name

  • Click on “Save”

    • in the pop-up window, specify a site name for this connection, e.g., “shark1” or “shark2” and then click “OK” to confirm

 

Let us start a connection by clicking on the site name (here as an example “ alice2”) and then on the button “Login”

You can always edit an existing “site” by clicking on the site name in the WinSCP Login screen and then on the “Edit” button.

Let us start a connection by clicking on the site name (here as an example “ shark2”) and then on the button “Login”

You can always edit an existing “site” by clicking on the site name in the WinSCP Login screen and then on the “Edit” button.

 

WinSCP will connect you first to the ssh gateway.

If you do this for the first time, WinSCP will ask you to confirm the authenticity of the host (see left image as an example) with “Yes”.

After you have confirmed, it will show you the login message from the gateway. Click “Continue”.

Afterwards, it will ask you for your ALICE user password:

If the authentication was successful, it will try to connect to the login node

If you set up WinSCP to use the LUMC ssh gateway, it will first establish a connection to it

If you do this for the first time, WinSCP will ask you to confirm the authenticity of the host (see left image as an example) with “Yes”.

Afterwards, it will ask you for your SHARK user password:

If the authentication was successful, it will try to connect to the login node

 

 

 

If it is the first time that you try to connect to one of the login nodes with WinSCP, it will ask you to confirm the authenticity of the server (see left image as an example)

Then, it will ask you once more for your ALICE user password:

If the authentication was successful, it will establish the connection

If it is the first time that you try to connect to one of the login nodes with WinSCP, it will ask you to confirm the authenticity of the server (see left image as an example).

Then, it will ask you once more for your SHARK user password:

If the authentication was successful, it will establish the connection

Now, you should see window split in the middle where the left side shows you directories on your local workstation and the right side, your home directory on ALICE.

Now, you should see window split in the middle where the left side shows you directories on your local workstation and the right side, your home directory on SHARK.

After you have established a connection, you can move through the directory structure on both sides until you find the data that you want to copy.

You can copy files or directories by dragging them from one side to the other which will initiate the copying process and show you the progress in a pop-up window.

You can also synchronise the content of the directories on both sides with the button “Synchronize”. This will open a pop-up window where you can set specific options for how you want to synchronise your data and also change the paths on your local workstation and the cluster which should be synchronised.

FileZilla

You can download FileZilla from the vendor website at https://filezilla-project.org .

Make sure to download FileZilla client. Once the installation is complete you will need to create an HPC profile. You can do so by going to the Site-Manager:

Create a new site and enable these settings:

ALICE

SHARK

  • Protocol: SFTP – SSH File Transfer Protocol

  • Host: res-hpc-lo02.researchlumc.nl or res-hpc-lo04.researchlumc.nl

  • Logon Type: Interactive

  • User: Your SHARK username

After the General tab settings have been filled out, select the Transfer Settings tab:

  • Check Limit number of simultaneous connections

  • Maximum number of connections: 1

These settings will keep a single connection open so you will not have to re-authenticate.

Cyberduck

You can download Cyberduck from the vendor website at Cyberduck | Libre server and cloud storage browser for Mac and Windows with support for FTP, SFTP, WebDAV, Amazon S3, OpenStack Swift, Backblaze B2, Microsoft Azure & OneDrive, Google Drive and Dropbox .

See the documentation for how to setup Cyberduck.

Like WinSCP, Cyberduck can handle tunnelling through an ssh gateway to establish an connection with sftp.