How to Transfer Files on TSCC

Several tools are available on TSCC to assist users in moving their data between filesystems and other resources. For files over 2GB in size, a great option is bbFTP, which uses multi-threading to maximize throughput over the 1-gigabit network connection. Tests have achieved between 60MB and 100MB per second using this method.

Other methods include scp, sftp and bbcp. Documentation for scp and sftp can be found in the Linux man pages. Windows users may prefer WinSCP for SSH, SCP and SFTP and PuTTY for SSH sessions. Those tools are available from WinSCP. Mac users can run the commands directly in the Terminal app built into Apple OS X.

Data Mover Node Now Available

A new host is available on the TSCC to give users a place to conduct large file copies without tying up a login node or spending cycles for a compute node. Its designation is tscc-dm1. Users can connect via ssh either from a TSCC login node or from outside (as tscc-dm1.sdsc.edu). Oasis, home directories, and project mounts are all mounted on this system.

As with the login nodes, please do not run any significant analyses on tscc-dm1. Parts of the TSCC software stack are missing from this node, and because of architecture differences, much of the application software that is installed isn't guaranteed to work correctly.

Note

Remote access tools are mentioned here for informational purposes only — users are responsible for configuration of remote access on their personal systems.

Overview of bbFTP

To use bbFTP, you need both a client and a server.

Client — a client is installed on TSCC at /opt/bbftp/bin
Server — the service is available on TSCC at
- tscc-login.sdsc.edu
- tscc.sdsc.edu

This server is installed in /opt/bbftp/bin. Use the -s option to invoke a remote host for bbftpd. You will be prompted for a password on this host (see example below).

Please Note

We have experienced problems using the Korn shell in this context. We recommend tcsh.

Local bbFTP Client Installation

If you prefer, you can use a local client on another machine or install your own bbFTP client. Here are the basic steps to do that:

Download the bbFTP client source
Extract the source code
Change to the bbftp directory
Run ./configure
Run make
Run make install

The default target location will be /user/local/bin so if you are unable to write that location or want to preserve an existing copy there, you should change the target for make install.

Additonal versions and server code are available for download from the bbFTP home page.

Basic bbFTP Syntax

bbftp

displays a brief help message

bbftp -v

displays version info

bbftp [Options] [-u RemoteUsername] -i ControlFile [RemoteHost]

bbftp [Options] [-u RemoteUsername] -e ControlCommands [RemoteHost]

requests the execution of commands contained in the control file ControlFile or the ControlCommands command string, using RemoteUsername on RemoteHost as the service

See the bbFTP documentation for complete details.

Example session for bbFTP

Here is an example of a bbFTP session. In this example, the file HOMA.tar is being copied from GPFS-WAN to the ariel user's /scratch file space on TSCC, using the remote bbFTP daemon from tscc-login2.

[someuser@login-2-38.local TEMP]% /opt/bbftp/bin/bbftp -s -e 'get /gpfs/projects/someuser/KRAKEN/HOMA.tar /scratch/s1/tscc/ariel/TEMP/HOMA.tar' -u someuser -p 2 -V tscc-login.sdsc.edu

Password: <someuser's password on tscc-login>

>> COMMAND : get /gpfs/projects/someuser/KRAKEN/HOMA.tar /scratch/s1/tscc/ariel/TEMP/HOMA.tar

<< OK

4299427840 bytes got in 36.7 secs (1.14e+05 Kbytes/sec or 894 Mbits/s)