Network Storage Services

We have started offering a network storage service to expand our research data storage environment and data collection infrastructure from May 2022. You can start using the service by following the procedure from User Portal. Please note that the maintenance may be performed and service content may be revised without prior notice.

  • Research data sharing by storage linkage with GakuNin RDM
  • Data collection for IoT devices, etc.
  • Back up of the storage of the supercomputer

Small network storage cluster connected to a network independent from the supercomputer system.

Item Content
Device DELL EMC isilon A200 x 4
DELL EMC PowerScale F200 x 4
Storage capacity 260TB
Front-end network 5Gbps

If you have a user ID of Large-scale computer system, you can use the following services free of charge. This service does not back up the user's data by the system administrator.Please make backups as necessary to prepare for the loss of the registered data when using this service.

Item Content
Supported protocols FTPS and S3 (Amazon S3 compatible, path format)
Storage capacity 100GB
Transfer capacity Unlimited
Backup None
FTP account 1
S3 account 1
Number of S3 Buckets 1
Number of S3 requests Unlimited
Number of S3 objects Unlimited
Tine zone of Device UTC

  1. Access the User Portaland log in with your user ID of Large-scale computer system.　
  2. Click "Network Storage Service" in the left column. portal_afterlogin
  3. Select "Generate (Regenerate)" from the drop-down list surrounded by red when issuing a password used for FTP or blue when issuing a secret key for S3, and agree to the precautions. Then click the "Submit" button. request_passwd
  4. When the password/secret key is successfully issued, the password information will be displayed at the top of the screen. Please note that the password information is displayed only once. generate_passwd

Item Content
Connection Destination s3.kudpc.kyoto-u.ac.jp
Access key Please check the user portal.
Secret key Please check the user portal.
Bucket name Please check the user portal.

Please refer to the Official Manual. The supercomputer system provides module files. Please see here for details.

$ aws configure --profile kudpc
AWS Access Key ID [None]:     ## Enter the access key issued by the User Portal
AWS Secret Access Key [None]: ## Enter the secret access key issued by the User Portal
Default region name [None]:   ## Enter without input
Default output format [None]: ## Enter without input

# Basic command system
$ aws --profile kudpc --endpoint-url=https://s3.kudpc.kyoto-u.ac.jp s3 [Command]

## Obtain a list of files
$ aws --profile kudpc --endpoint-url=https://s3.kudpc.kyoto-u.ac.jp s3 ls s3://b59999

## Upload Files
$ aws --profile kudpc --endpoint-url=https://s3.kudpc.kyoto-u.ac.jp s3 cp sample.sh s3://b59999

## Download Files
$ aws --profile kudpc --endpoint-url=https://s3.kudpc.kyoto-u.ac.jp s3 cp s3://b59999/sample.sh ./

## Delete files from network storage
$ aws --profile kudpc --endpoint-url=https://s3.kudpc.kyoto-u.ac.jp s3 rm s3://b59999/sample.sh

## Synchronize local folders to network storage
$ aws --profile kudpc --endpoint-url=https://s3.kudpc.kyoto-u.ac.jp s3 sync ./ s3://b59999

Please refer to the Official Manual. The supercomputer system can be used without loading the module.

## Create directory to mount network storage
$ mkdir ./kudpc-s3

## Create S3 authentication information
$ echo "{accesss key}:secret key}" > ${HOME}/.passwd-s3fs
$ chmod 600 ${HOME}/.passwd-s3fs

## mount
$ s3fs b59999 ./kudpc-s3 -o use_path_request_style,url=https://s3.kudpc.kyoto-u.ac.jp,uid=$(id -u),gid=$(id -g)
## unmount
$ fusermount -u ./kudpc-s3

Please download the file from official website.

  1. Start Cyberduck.
  2. Click "New Connection" at the top of the screen.
  3. Enter the session information as shown in the table below and click the "Connect" button.
Item Content
Protocol Selection Screen Amazon S3
Server s3.kudpc.kyoto-u.ac.jp
Port 443
Access key ID S3 access key
Secret access key S3 secret key

Cyberduck_S3Connect

Item Content
Connection Destination s3.kudpc.kyoto-u.ac.jp
Account name Please check the user portal.
Password Please check the user portal.

Please refer to Installing WinSCP.

  1. Start WinSCP.
  2. Please enter your session information as shown in the table below and click the "Login" button.
Item Content
Transfer protocol FTP
Encryption Explicit TLS/SSL encryption
Host name s3.kudpc.kyoto-u.ac.jp
Port number 21
User name FTP account name
Password FTP Password

WinSCP_FTPinit

  1. If successful, the WinSCP screen will be displayed and file transfer will be possible.

Please download the file from official website.

  1. Start Cyberduck.
  2. Click "New Connection" at the top of the screen.

  3. Enter the session information as shown in the table below and click the "Connect" button.

    Item Content
    Protocol Selection Screen FTP(File Transfer Protocol)
    Server s3.kudpc.kyoto-u.ac.jp
    Port 21
    User name FTP account name
    Password FTP Password

    Cyberduck_FTPinit