File Transfer with Archaea

We are providing Archaea (Former HCP Tools) as a high-speed file transfer tool.

Archaea (Former HCP Tools) uses the HpFP protocol, which enables fast file transfer. You can use it by installing the dedicated package.

  • The installation of Archaea (Former HCP Tools) must be completed by following the procedure in Installing Archaea(HCP Tools) client.
  • Generate a key pair by following the procedure in Generating key pair and registering public key, and register a public key via User Portal.
  • The private key in RSA format created in the above procedure must be placed on the following path.
    ・When using Windows as a client：
　　~/_hcp/id_rsa
・When using Linux as a client：
　　~/.hcp/id_rsa

    The private key storage location can be changed in the configuration file. Please refer to theOnline Documents for details.

System Name Host Name Note
File Transfer Server hpcfs.kudpc.kyoto-u.ac.jp Consists of two servers.

Please specify File Transfer Server(hpcfs.kudpc.kyoto-u.ac.jp) as the connection destination. You cannot connect to System B(laurel.kudpc.kyoto-u.ac.jp), etc.

This section explains how to log in to the supercomputer systems using the hcp command. The hcp command uses the command prompt (terminal).

Upload  : hcp [OPTION]... SOURCE [SOURCE]... [USER@]HOST[:PORT]:DEST
Download : hcp [OPTION]... [USER@]HOST[:PORT]:SOURCE [:SOURCE]... DEST

Short options Long options Description
-p --permission Preserve permissions of the transfer source.
-R --recursive Copy all files under each directory recursively.
　 --hpfp Use HpFP protocol.
　 --user= Specify user name.
　 --mcd= Specify the number of multiple connections.
-h --help Display help.

In the following examples, user ID will be 'b59999'.
Please open a command prompt (terminal) on your PC to perform the following operations.

  • Making a copy of “file1.txt” to a remote machine. 
    ```nohighlight
$ hcp file1.txt hpcfs.kudpc.kyoto-u.ac.jp:~/
Login as:   ←Entering your user ID
Passphrase for the key:   ←Entering your passphrase.
  • Specifying user ID, and making a copy of “file1.txt” to a remote machine. 
    $ hcp --user=b59999 file1.txt hpcfs.kudpc.kyoto-u.ac.jp:~/
Passphrase for the key:   ←Entering your passphrase.
or
$ hcp file1.txt b59999@hpcfs.kudpc.kyoto-u.ac.jp:~/
Passphrase for the key:   ←Entering your passphrase.
  • Making a copy of “file1.txt” on a remote machine as “file2.txt.” to a local machine. 
    $ hcp hpcfs.kudpc.kyoto-u.ac.jp:file1.dat file2.dat 
Login as:   ←Entering your user ID.
Passphrase for the key:   ←Entering your passphrase.
  • Making a copy of a directory “dir1” to a remote machine. 
    $ hcp -R dir1 hpcfs.kudpc.kyoto-u.ac.jp:~/
Login as:   ←Entering your user ID.
Passphrase for the key:   ←Entering your passphrase.
  • Using HpFP protocol to make a copy of “file1.txt” to a remote machine. 
    $ hcp --hpfp file1.txt hpcfs.kudpc.kyoto-u.ac.jp:~/
Login as:   ←Entering your user ID.
Passphrase for the key:   ←Entering your passphrase.
  • Specifying a number of multiple connections, and making a copy of “file1.txt” to a remote machine. 
    $ hcp --mcd=1 file1.txt hpcfs.kudpc.kyoto-u.ac.jp:~/
Login as:   ←Entering your user ID.
Passphrase for the key:   ←Entering your passphrase.

    ※The mcd (number of multiple connections) is set to 8 by default.

For detailed options and descriptions of options for other commands (hcp-ls, hmv, etc.), please check the --help option or refer to the Online Documents.

  1. Start the Arcaea dialog.
  2. Double-click 'New site'. dialog_1
  3. Enter the following information and click 'Private key management'. FQDN or IP address：hpcfs.kudpc.kyoto-u.ac.jp User name: User number (ID) Authentication method: public key authentication Protocol and security: TCP or HpFP dialog_2
  4. Enter the passphrase and click on OK. dialog_3
  5. Click on 'OK' when the message 'The import of the private key was successful' appears. dialog_4
  6. Confirm that the key that you have just registered appears in the list of registered keys and close the screen. dialog_5
  7. From the list of sites, select the site you have just set up and click on 'Connect'. (From the second time onwards, start from here). dialog_6
  8. Enter the passphrase of the registered key and click on 'OK'. If successful, the file list screen will be displayed and file transfer wil be enabled. dialog_7 dialog_8

HpFP is a UDP-based protocol. In the environments where delay/packet loss occurs, HpFP (--hpfp option) can be used to speed up the process. If HpFP (--hpfp option) is not used, the connection is made via TCP. It is somewhat faster than scp or sftp transfers due to session multiplexing.

Communication via the WebSocket protocol is not permitted.

For communication with HCP tools, 874/tcp and 65520/udp (with the --hpfp option) is used. Depending on your environment, the port may not be open. In this case, communication is not possible.

To ensure fair use of the file transfer server, the number of connections to the server per user is limited to '16'. An example of the number of simultaneous connections when the number of connections is specified using the -mcd option is as follows.

  • If the --mcd option is not specified
    The number of connections is deemed to be 8 and the maximum number of simultaneous connections will be 2. *If 8 is specified in the --mcd option
    The number of connections is deemed to be 8 and the maximum number of simultaneous connections will be 2.
  • If 1 is specified in the --mcd option
    The number of connections is deemed to be 1 and the maximum number of simultaneous connections will be 16.

  • In case key or passphrase is wrong 
    Authentication failed since performing it with a challenge and the user inputs is not successful.
Authentication was failed.

    Make sure that the key or passphrase is correct.

  • In case some of the multiple connections fail 
    CH[12, primary=0] setup failed.
Error code (9, Bad file descriptor)

    The connection that has been successfully negotiated will be used for communication. Please wait until communication is completed.

  • In case referring to an no permission directory 
    A signal message was received from the remote host. The session will abort in a reason specified by the message (reason=0xA003).
Could not offer a request @ _hcp::proto::HcppSessionInitiator_:L177.

    Make sure that the path to the directory is correct.

A record of the execution of an HCP Tools command is recorded in the following file in the execution directory if the -hcp-out option does not specify a file to output.

・When Windows is used as a client：
　　.hcp.out
・When Linux is used as a client：
　　_hcp.out

SRC test.txt
DST hpcfs.kudpc.kyoto-u.ac.jp:65520:/home/b/b59999/
OK 0000 FT 00000001 /home/b/b59999/test.txt
EXIT 0 REASON 0000

The command has terminated successfully (EXIT 0).

SRC test.txt
DST hpcfs.kudpc.kyoto-u.ac.jp:65520:/home/b/b59999/
OK 0000 FT 00000001 /home/b/b59999/test.txt
EXIT 72 REASON 9008

The command has terminated with an authentication failure (EXIT 72).

Please refer to Online Documents for details on the exit codes (command exit status).

In the current version (Archaea dialog 1.0), there is no function to specify and move a directory path directly. The function will be implemented in the future.
The ability to move by specifying a directory path has been implemented. dialog_9