Transferring Files With HCP tools (on a trial basis)

We are providing HCP Tools as a high-speed file transfer tool.

HCP Tools uses the HpFP protocol, which enables fast file transfer. HCP Tools can be used by installing the dedicated package.

  1. The installation of HCP tools must be completed by the procedure of Installing HCP tools.
  2. Generate a key pair by the procedure of Generating Key Pair and Registering Public Key,and then register a public key from User Portal.
  3. The private key in RSA format created in the above procedure must be placed in the following path.
    The location of the private key file can be changed in the configuration file. Please refer to the reference manual for more details.
    ・When using Windows as a client:
      ~/_hcp/id_rsa
    ・When using Linux as a client:
      ~/.hcp/id_rsa

    The private key storage location can be changed in the configuration file. Please refer to the reference manual for details.

This section explains how to log in to the Supercomputer Systems using the hcp command. The hcp command uses the command prompt (terminal).

Upload : hcp [OPTION]... SOURCE [SOURCE]... [USER@]HOST[:PORT]:DEST
Download : hcp [OPTION]... [USER@]HOST[:PORT]:SOURCE [:SOURCE]... DEST

Short options Long options Purposes
-p --permission Preserve information about file permissions of the transfer source
-R --recursive Copy all files under each directory, recursively
  --hpfp Using HpFP protocol.
  --user=<username> Specifying user name.
  --mcd=<number> Specifying a mcd for connections.
-h --help Display help.

In the following examples, user ID will be “b59999.”
Please open a command prompt (terminal) on your PC to perform the following operations.

  • Making a copy of “file1.txt” to a remote machine.
    $ hcp file1.txt hpcfs.kudpc.kyoto-u.ac.jp:~/
    Login as:   ←Entering your user ID.
    Passphrase for the key:   ←Entering your passphrase.
  • Specifying user ID, and making a copy of “file1.txt” to a remote machine.
    $ hcp --user=b59999 file1.txt hpcfs.kudpc.kyoto-u.ac.jp:~/
    Passphrase for the key:   ←Entering your passphrase.
    or
    $ hcp file1.txt b59999@hpcfs.kudpc.kyoto-u.ac.jp:~/
    Passphrase for the key:   ←Entering your passphrase.
  • Making a copy of “file1.txt” in a remote machine as “file2.txt.” to a local machine.
    $ hcp hpcfs.kudpc.kyoto-u.ac.jp:file1.dat file2.dat 
    Login as:   ←Entering your user ID.
    Passphrase for the key:   ←Entering your passphrase.
  • Making a copy of a directory “dir1” to a remote machine.
    $ hcp -R dir1 hpcfs.kudpc.kyoto-u.ac.jp:~/
    Login as:   ←Entering your user ID.
    Passphrase for the key:   ←Entering your passphrase.
  • Using HpFP protocol to make a copy of “file1.txt” to a remote machine.
    $ hcp --hpfp file1.txt hpcfs.kudpc.kyoto-u.ac.jp:~/
    Login as:   ←Entering your user ID.
    Passphrase for the key:   ←Entering your passphrase.
  • Specifying a number of multiple connections, and making a copy of “file1.txt” to a remote machine.
    $ Specifying a number of multiple connections, and making a copy of “file1.txt” to a remote machine.
    hcp --mcd=1 file1.txt hpcfs.kudpc.kyoto-u.ac.jp:~/
    Login as:   ←Entering your user ID.
    Passphrase for the key:   ←Entering your passphrase.

    ※The mcd (number of multiple connections) is set to 8 by default.

See the --help option or the command reference for detailed options and descriptions of options for other commands (hcp-ls, hmv, etc.).

  • In case key or passphrase is wrong
    2022/02/10 10:45:49 0000498c:INFO :Authentication failed.
    2022/02/10 10:45:49 0000498c:INFO :Authentication was failed in building a point.hcp::node:HcpnException @ hcp::node::HcpnEndPointTransfer:L454 :  > hcp::proto::HcppException @ hcp::proto::HcppSession:L1214 :

    Make sure that the key or passphrase is correct.

  • In case some of the multiple connections fail
    2022/01/06 16:30:22 00007fc882ffd700:INFO :CH[6, primary=0] negotiate failed.
    2022/01/06 16:30:22 00007fc882ffd700:INFO :hcp::proto::HcppException @ hcp::proto::HcppSessionSecondaryChannel:L1148 : No welcome message from the server was received. The server might refuse your request to connect subject to its admission control or access control list.

    The connection that has been successfully negotiated will be used for communication. Please wait until communication is completed.

  • In case referring to an no permission directory
    2022/01/12 13:50:12 00007f9c175c8700:INFO :A signal message was received from the remote host. The session will abort in a reason specified by the message (reason=0xBFFB).
    2022/01/12 13:50:12 00007f9c15dc5700:INFO :Could not offer a request @ hcp::proto::HcppSessionInitiator_:L172.

    Make sure that the path to the directory is correct.

  • When a congestion control mode that is not allowed is specified
    2022/01/12 17:18:22 00002930:INFO :Your requested congestion mode of HpFP will not be used (requested=AGGRESSIVE, actual=FAIR).

    Connect in FAIR mode.

HpFP is a UDP-based protocol. In an environment where latency and packet loss are occurring, using HpFP (--hpfp option) is expected to speed up the process.
When HpFP (--hpfp option) is not used, TCP is used for the connection. Because of session multiplexing, it is slightly faster than scp or sftp.

Communicating with the WebSocket protocol is not allowed.

For communication with HCP tools, use 874/tcp and 65520/udp (when the --hpfp option is used).
Depending on your environment, the port may not be open. In that case, communication is not possible.

For fair use of the file transfer server, the number of server connections per user is limited to 8. By default, the number of connections to the server used for file transfer is set to "8" to speed up file transfer, and multiple hcp commands cannot be executed simultaneously.To execute multiple hcp commands simultaneously, the number of connections must be specified using the --mcd option.

  • In case the --mcd option is not specified
    The number of connections is considered to be 8, and the maximum number of simultaneous connections is 1.
  • In case 8 is specified in the --mcd option
    The number of connections is considered to be 8, and the maximum number of simultaneous connections is 1.
  • In case 1 is specified in the --mcd option
    The number of connections is considered to be 1, and the maximum number of simultaneous connections is 8.

The congestion control mode allows only FAIR.
If you specify a mode other than FAIR, a message "When you specify a congestion control mode that is not allowed" is displayed and the connection is made with FAIR.

In HCP tools version 1.3.0R2_49 or later, which is currently available, the functionality has been enhanced to apply auxiliary groups to determine file access permissions. This makes it possible to transfer files on a large volume disk (LARGE) without being aware of the default group at login.


Copyright © Academic Center for Computing and Media Studies, Kyoto University, All Rights Reserved.