Tips for Job Execution

Environment Variables

Setting and Reference of the Environment Variables

Use export command to set environment variables and add "$" at the beginning of variable names to refer to the environment variables.

• Setting of the Environment Variables

  #Format　Environment Variable Name=Value; export Environment Variable Name
  LANG=en_US.UTF-8; export LANG

• Referene of the Environment Variables

  echo $LANG

Environment variables set when executing jobs

Options not available when submitting jobs

Please note that the following options cannot be specified following #SBATCH in the job script.

/tmp Area

The /tmp area can be used as a temporary data write destination on our supercomputer system. In some cases, programs with detail file I/O can be processed faster by using /tmp.

/tmp is a private area for each job so that files are not mixed with those of other jobs. Please take advantage of this feature to avoid mixing files with those of other jobs.

Please note that the /tmp area is automatically deleted at the end of the job. To keep the files written to /tmp, you must include a description to copy the files to /home or /LARGE0,/LARGE1 in the job script . Please note that the deleted files cannot be retrieved later.

Specific Examples of using /tmp

• Specify /tmp as the write destination of the file if programmatically possible to specify.
• Place files to be read repeatedly in /tmp before program execution starts.
• Place programs and input files that access files by relative PATH in /tmp and execute.

Available /tmp capacity

/tmp capacity available for each system is calculated from Number of processes x number of cores per process x capacity per core per system (see the following table)

For example, if a job with 4 processes (8 cores per process) is submitted on System B, from 4 x 8 x 8.9, 284.8GB will be allocated.

Specify the order of the job execution

It is possible to control the timing of the job execution start to be submitted according to the execution status of jobs that have already been submitted. Specifically, it is available by adding the following information to the job submission options column.

#SBATCH -d <Commands specifying the execution order>:<Job ID already submitted>

For example, if job script A has been successfully completed execution and you wish to continue executing job script B, you can use the following procedure.

1. Submit job script A to the job.

2. If the job ID of the job submitted in 1. is "200", job script B will be as follows. *

   #!/bin/bash
   #============ SBATCH Directives =======
   #SBATCH -p gr19999b
   #SBATCH -t 2:0:0
   #SBATCH --rsc p=4:t=8:c=8:m=8G
   #SBATCH -d afterok:200
   #SBATCH -o %x.%j.out
   #============ Shell Script ============
   srun ./a.out

3. Submit job script B to the job.

You can select the specified command of the execution order from the following four options.

Execution of array jobs

You can execute multiple jobs with different parameters in a single job script with the function of the array job. Specifically, it is available by adding the following information to the job submission options column.

#SBATCH -a <start_num>-<end_num>[option]

For example, you can execute a job to transfer 1.data, 2.data, and 3.data placed in the same directory as the job script to ./a.out and analyze by writing a following job script.

#!/bin/bash
#============ SBATCH Directives =======
#SBATCH -p gr19999b
#SBATCH -t 2:0:0
#SBATCH --rsc p=4:t=8:c=8:m=8G
#SBATCH -a 1-3
#SBATCH -o %x.%A_%a.out
## %x is replaced by the job name, %A by the array job ID, and %a by the array task ID.
#============ Shell Script ============
srun ./a.out ${SLURM_ARRAY_TASK_ID}.data

The following options can be set in the [option] field.

How to execute multiple programs simultaneously on a single computing node

You can use computing resources effectively by executing multiple programs simultaneously as a single job. You can execute multiple programs simultaneously as a single job by describing multiple execution commands in a shell script and executing the shell script.

※This method is for a sequential program or a program that is thread-parallelized by OpenMP or automatic parallelization functions. ※To execute multiple MPI programs simultaneously as a single job, please refer to How to execute multiple programs simultaneously (MPMD).

An example of each script is shown below.

Sequential Execution (1 thread per program)

Job scripts to be executed by sbatch command (sequential)

※The following script is a sample of executing four programs simultaneously.

#!/bin/bash
#============ SBATCH Directives =======
#SBATCH -p gr19999b
#SBATCH --rsc p=4:t=1:c=1:m=3G
#SBATCH -o %x.%j.out
#============ Shell Script ============
srun ./multiprocess.sh

• Specify the queue with the -p option.
• Specify the resource to be used with the --rsc option as follows.
  • The argument p is the number of processes to be used.
    • Specify the number of programs you wish to execute simultaneously. In this case, the number is 4.
  • The argument c is the number of cores to be used per process (program).
    • Since the program is a single-threaded execution, specify 1.
  • The argument t is the number of threads to be used per process (program).
    • Since the program is a single-threaded execution, specify 1.
  • The argument m is the amount of memory per memory process (program) to be used.
• Describe the path of the shell script to be executed following the srun command.

Shell scripts to be executed in job scripts (sequential)

#!/bin/bash
case $SLURM_PROCID in
    0) ./a1.out ;;
    1) ./a2.out ;;
    2) ./a3.out ;;
    3) ./a4.out ;;

esac

• When resources are allocated, a rank number is assigned to each process.
• When the process is executed, it is possible to read the assigned rank number from the environment variable SLURM_PROCID.
• By creating shell scripts that use rank numbers to branch processing, it is possible to separate the programs to be executed by process. This allows multiple programs to be executed simultaneously as a single job.

OpenMP Execution

Job script to be executed by sbatch command (OpenMP)

※The following script is a sample to execute 8 programs with 4 threads execution simultaneously.

#!/bin/bash
#============ SBATCH Directives =======
#SBATCH -p gr19999b
#SBATCH -t 2:0:0
#SBATCH --rsc p=8:t=4:c=4:m=3G
#SBATCH -o %x.%j.out
#============ Shell Script ============
srun ./multiprocess.sh

• Specify the queue with the -p option.
• Specify the resource to be used with the --rsc option as follows.
  • The argument p is the number of processes to be used.
    • Specify the number of programs you wish to execute simultaneously. In this case, the number is 8.
  • The argument c is the number of cores to be used per process (program) .
    • The program is a 4-thread execution, so 4 is specified.
  • The argument t is the number of threads to be used per process (program) used.
    • The program is a 4-thread execution, so 4 is specified.
  • The argument m is the amount of memory per process (program) to be used.
• Describe the path of the shell script to be executed following the srun command.

Shell scripts (OpenMP) to be executed in job scripts

#!/bin/bash
case $SLURM_PROCID in
    0)  ./aaa.out ;;
    1)  ./bbb.out ;;
    2)  ./ccc.out ;;
    3)  ./ddd.out ;;
    4)  ./eee.out ;;
    5)  ./fff.out ;;
    6)  ./ggg.out ;;
    7)  ./hhh.out ;;

esac

• When resources are allocated, a rank number is assigned to each process.
• When the process is executed, it is possible to read the assigned rank number from the environment variable SLURM_PROCID.
• By creating shell scripts that use rank numbers to branch processing, it is possible to separate the programs to be executed by process. This allows multiple programs to be executed simultaneously as a single job.

By specifying "#SBATCH --rsc t=4" in the job script, the job scheduler is automatically configured to use 4 threads per process. Therefore, in the above shell script, each program is executed with 4 threads.

If you want to vary the number of threads per process, define OMP_NUM_THREADS={number of threads you want to use} for each process as follows.

#!/bin/bash
case $SLURM_PROCID in
    0) export OMP_NUM_THREADS=1; ./aaa.out ;;
    1) export OMP_NUM_THREADS=2; ./bbb.out ;;
    2) export OMP_NUM_THREADS=2; ./ccc.out ;;
    3) export OMP_NUM_THREADS=3; ./ddd.out ;;
    4) export OMP_NUM_THREADS=3; ./eee.out ;;
    5) export OMP_NUM_THREADS=4; ./fff.out ;;
    6) export OMP_NUM_THREADS=4; ./ggg.out ;;
    7) export OMP_NUM_THREADS=4; ./hhh.out ;;

esac

How to execute multiple programs simultaneously (MPMD)

Job script to be executed by sbatch command (MPI)

※The following script is a sample to execute 3 MPI program (4 cores and 4 threads per process) that requires 4 processes concurrently.

#!/bin/bash
#============ SBATCH Directives =======
#SBATCH -p gr19999b
#SBATCH -t 2:0:0
#SBATCH --rsc p=12:t=4:c=4:m=3G
#SBATCH -o %x.%j.out
#============ Shell Script ============
srun --multi-prog multiprocess.conf

• Specify the queue with the -p option.
• Specify the resource to be used with the --rsc option as follows.
  • The argument p is the number of processes to be used.
    • Specify the number of programs you want to execute concurrently.(If you want to execute 3 MPI programs that use 4 processes concurrently, specify 12 from 4 x 3)
  • The argument c is the number of cores to be used per process (program) .
    • The program is a 4-thread execution, so specify 4.
  • The argument t is the number of threads to be used per process (program) used.
    • The program is a 4-thread execution, so specify 4.
  • The argument m is the amount of memory per process (program) to be used.
• The srun command is specified with the following options.
  • In the multi-prog, enter the path to the configuration file (multiprocess.conf) describing the execution of the MPI program to be created below.

configuration file(multiprocess.conf)

## Execute as a 4-process MPI program
0-3  ./aaa.out
4-7  ./bbb.out
8-11 ./ccc.out

• In the first column, enter the range of process ID used by the MPI program. (Note that process ID must begin with zero.)
• In the second column, enter the path (absolute path or relative path) of the MPI program to be executed with the process ID specified in the first column.

The number of processes used can be changed for each MPI program. For example, if you want the program a1.out to use 4 processes and a2.out to use 8 processes, you can create the following configuration file. However, the number of cores, threads, and memory per process are automatically set to the values specified by the --rsc option, so they cannot be customized for each MPI program.

## Execute as a 4-process MPI program
0-3  ./aaa.out
## Execute as a 8-process MPI program
4-11 ./bbb.out