Using File System

All users can use each home directory. Large volume disk (LARGE) is available in the Personal Course, Group Course, and Private Cluster Course. Use large volume disk (LARGE) when dealing with a large-scale data in the Personal Course, Group Course, and Private Cluster Course.

The home directory path is shown below:

  • A capacity of up to 100 GB is available. (User Quota)
  • The “b” in the path is a placeholder for the first letter of the user ID.

Large volume disk (LARGE) is available in the Personal Course, Group Course, and Private Cluster Course. Large volume disk consists of the pair of /LARGE0 and /LARGE1, and the pair of /LARGE2 and /LARGE3. Either pair is assinged for each personal course user and group. Which pair your course has been assigned is written in the service approval email.

LARGE configuration for personal course is shown below.

  • A total capacity of 2TB is available. (Group Quota)
    • It is divided into two areas.
  • In the initial setting, one area is configured for backup.
    • You can make backup settings. For how to set up backups, see Backup Setting.

LARGE configuration for group and private cluster course is shown below.You can create and use your own directory into a directory for the group.

  • The capacity meets the computing resources that you applied for.
    • LARGE restricts the capacity in Group Quota.
    • It is divided into two file systems.
  • You need to change the current group when putting a file in the LARGE storage. For how to change the current group, see Changing Current Group
  • A group manager can change a setting to use a backup area as the normal area. For how to set up, see information for Group Managers.

In the system A/B/C, a structure called Burstbuffer is connected for the purpose of speeding up the file access. System A has the DataWarp of Cray Inc. and system B,C has the IME of DDN Inc. Registration is needed to use the burstbuffers. Please see the details below.

The quota command allows to check the current disk use status.

  • blocks: File capacity currently in use (KB)
  • quota: File capacity/number limit (software limit)
  • limit: File capacity/absolute number limit (hardware limit)
  • grace: Allowable duration of limit violence (exceeding software limit)
  • files: A number of files currently in use

$ quota -u b59999
Disk quotas for user b59999 (uid 12345):
  Filesystem       blocks        quota        limit grace    files    quota    limit grace
       /home     17859980     30000000     31000000     -    69218   600000   700000     -
     /LARGE0           16            1            1     -        4        1        1     -
     /LARGE1            0            1            1     -        0        1        1     -

LARGE system restricts the capacity in Group Quota. You can execute the quota command with the -g option.

$  quota -g
Disk quotas for group gr19999 (gid 19999):
  Filesystem       blocks        quota        limit grace    files    quota    limit grace
       /home       151808            0            0     -       22        0        0     -
     /LARGE0   2088651052   8000000000   8050000000     -    90543 16000000 16100000     -
     /LARGE1   1967559464   8000000000   8050000000     -    90544 16000000 16100000     -

In the group course and private cluster course, before using the large volume disk(LARGE) or creating a file with group authority, please change the current group by the newgrp command

By default, files are created with the group permission set to user name. Such files are not counted as group's files. For users' convenience, users can create files with the group permission set to user name, but such files can be created only up to 1TB on the large volume disk.

In the group course and private cluster course, before using the large volume disk(LARGE) or creating a file with group authority, please change the current group by the newgrp command.

## Checking the current group of UID and GID.
$ id
uid=59999(b59999) gid=59999(b59999) groups=19999(gr19999),59999(b59999)

## Changing the current group to gr19999.
$ newgrp - gr19999
$ id
uid=59999(b59999) gid=19999(gr19999) groups=19999(gr19999),59999(b59999)

The chdefgrp command is also provided to change the default group.

## Changing the default group to gr19999.
$ chdefgrp gr19999

After using newgrp command, a part of the environment variables which were set by modules before using newgrp command could be disappeared. (e.g. LD_LIBRARY_PATH)

reload_module command is available to restore the environment variables set by modules after using newgrp command.

Example usage:

echo $LD_LIBRARY_PATH
/opt/app/intel/impi/2017.2.174/intel64/lib:/opt/app/intel/compilers_and_libraries_2017.2.174/linux/tbb/lib/intel64/gcc4.4:/opt/app/intel/compilers_and_libraries_2017.2.174/linux/mkl/lib/intel64:/opt/app/intel/compilers_and_libraries_2017.2.174/linux/compiler/lib/intel64:/opt/app/intel/compilers_and_libraries_2017.2.174/linux/ipp/../compiler/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/app/intel/compilers_and_libraries_2017.2.174/linux/mpirt/lib/intel64:/opt/app/intel/compilers_and_libraries_2017.2.174/linux/ipp/lib/intel64:/opt/cray/lib64:

newgrp gr19999

echo $LD_LIBRARY_PATH
/opt/cray/lib64:  # The content has been disappeared.

reload_module

echo $LD_LIBRARY_PATH
/opt/app/intel/impi/2017.2.174/intel64/lib:/opt/app/intel/compilers_and_libraries_2017.2.174/linux/tbb/lib/intel64/gcc4.4:/opt/app/intel/compilers_and_libraries_2017.2.174/linux/mkl/lib/intel64:/opt/app/intel/compilers_and_libraries_2017.2.174/linux/compiler/lib/intel64:/opt/app/intel/compilers_and_libraries_2017.2.174/linux/ipp/../compiler/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/app/intel/compilers_and_libraries_2017.2.174/linux/mpirt/lib/intel64:/opt/app/intel/compilers_and_libraries_2017.2.174/linux/ipp/lib/intel64:/opt/cray/lib64:

  • This command changes all default groups of the system A, B, and C.
  • Changes take around 10 minutes to be reflected.

Lustre is the parallel file system which consisted of a logical unit of multiple disk devices and I/O servers. By default, the stripe setting is disabled. Setting up striping enables a single process, a single file I/O. It can distribute the load between multiple disk devices, and help provide high-speed access.

Setting up striping

・Example: changing the stripe setting of the “st04” file (four-way stripe)

$ lfs setstripe -s 1048576 -o -1 -c 4 st04
Option Purpose
-c Specifies a number of stripes and a stripe width.
-o Specifies an offset.
-s Specifies a stripe size.

Displaying a stripe information

・Example: displaying a stripe information for the “st04” file

$ lfs getstripe -v st04


Copyright © Academic Center for Computing and Media Studies, Kyoto University, All Rights Reserved.