Allocation of Computing Resources

The supercomputer system provides a queue for each service course you have applied for, and you can specify the queue to execute the jobs.

All queues have three limits of computing resources for minimum, standard and maximum and the amount of computing resources available in the queue is determined.

Contents Meaning
Minimum Resources(Rmin) Number of CPU cores always secured for the queue
Standard Resources(Rstd) Maximum number of CPU cores that can use jobs submitted to the queue
Maximum Resources(Rmax) Maximum number of CPU cores available in the entire queue

The minimum resources are the number of CPU cores that are always secured for the queue and is also called the minimum guaranteed resources.
During regular system operation, the number of CPU cores set as the minimum resources are guaranteed, regardless of system congestion.

The standard resources are the maximum number of CPU cores that a submitted job to the queue can use.
You can specify the amount of resources to be allocated to a job by using the --rsc option when a job is submitted, but an error will occur if the resources which exceed standard resources are requested.

$ sbatch --rsc p=400 ./jobscript.sh
Too many processors requested. Job not submitted.

The maximum resources are the maximum number of CPU cores available in the entire queue. When system resources are available, jobs submitted to the queue are executed sequentially, and multiple jobs may be in the execution state at the same time. However, the sum of the number of cores used by all jobs in the queue cannot exceed the maximum amount of resources. When the number of CPU cores used by the entire queue reaches the maximum resources, the job will be in the waiting state even if the system resources are available.

【Amendment】

  • An amendment was made in the description of "Amount of memory per process" in System A. (2024/05/29)
    • Initial alue: (incorrect) 1142M -> (correct) 1071M
    • Maximum value: (incorrect) 120G -> (correct) 117G
Option Description System A
Initial Value
Maximum Value		 System B
Initial Value
Maximum Value		 System C
Initial Value
Maximum Value
p Number of processes 1 Standard Resources
(When c=1)		 1 Standard Resources
(When c=1)		 1 Standard Resources
(When c=1)
t Number of threads per process 1 112(224) 1 112(224) 1 112(224)
c Number of cores per process 1 112 1 112 1 112
m Amount of memory per process
(Unit：M, G)		 1071M 117G 4571M 500G 18392M 2011G


Option Description System G
Initial Value
Maximum Value		 Cloud
Initial Value
Maximum Value
p Number of processes 1 Standard Resources
(When c=1)		 1 1
t Number of threads per process 1 64(128) 1 36(72)
c Number of cores per process 1 64 1 36
m Amount of memory per process
(Unit：M, G)		 8000M 500G 14222M 500G
g Number of GPUs 1 Standard Resources - -

The value in () of t is the maximum value when hyper-threading is enabled. If you enable hyper-threading, please specify that t=c✕2.

  • If you do not specify the value of m, the initial value for the amount of memory per process✕ the number of cores per process is automatically set.
  • If a value of m is specified, it is assumed that the number of cores is required in proportion to it. Specifically, as follows.
    • If p=1:t=4:c=4:m=9142M (initial value 4571M x2) is specified for a job using System B, the c option requires 4 cores and the m option requires 2 cores. In this case, the number of cores of the option that allocates more cores is used as the value of c, resulting in a job with 4 cores per process. Although the job uses 4 cores per process, the maximum amount of memory is the specified resource (9142M).
    • If it is specified that p=1:t=2:c=2:m=13713M (initial value 4571M x3) in a job using system B, the c option require 2 cores and the m option require 3 cores. In this case, the number of cores required by the m option is larger than that required by the c option, resulting in 3 cores job per process.

If you use system G, you can require resources with either ptcm option or g option.

  • When you require resources using each ptcm option, 1 GPU per 16 cores is automatically allocated.
  • When you require GPU using g option, it will become as follows.
    • If g=1, the same parameter as p=1:c=16:m=128000M is automatically set as well as 1 GPU is allocated.
    • If g=2, the same parameter as p=2:c=16:m=128,000M is automatically set as well as 2 GPUs are allocated.
      • If you use 2 GPUs in one process, it is available by specifying -n 1 option to the srun command in the job script. (Example: srun -n 1 . /a.out)
      • Conversely, if you want to increase the number of processes, please adjust the number of cores per process with -c option of srun as well. (Example: srun -n 9 -c 1 . /a.out).

A group course queue is an individual queue for a group that have applied for a group course.

The amount of minimum resources, standard resources, and maximum resources are determined as follows

Contents Value
Minimum Resources Value calculated from course type and standard resources
Standard Resources Value indicated in the application form
Maximum Resources Varies in the range of 1 to 2 times of the standard resources

The large-scale job course queue is an individual queue for those who have applied for the large-scale job course.

The amount of minimum resources, standard resources, and maximum resources are determined as follows.

Contents Value
Minimum Resources Same value as the standard resources
Standard Resources Value indicated in the application form
Maximum Resources Same value as the standard resources

We explain the function of the --rsc option with an example. It is the same not only for the batch jobs but also for the --rsc option specified with tssrun.

We assume system B here, but the rule of the --rsc option is the same for other systems.

  • Specifications per node for system B
    • CPU：112 Core
    • Memory：4096M per core (500G for entire node)
    • Hyper-Threading：Up to 2 threads per core

  • p : Number of MPI processes. Default 1
  • t : Number of OpenMP threads per process. Default 1
  • c : Number of cores per process. Default 1
  • m : Amount of memory per process. Default 4096M

#SBATCH --rsc p=1:t=4:c=4:m=8G
  • It allocate one threads per physical core of the CPU and execute.
  • The available memory is 8 GB.

#SBATCH --rsc p=60:t=1:c=1
  • It allocate one process per physical core of the CPU and execute.
  • The available memory for each process is 4096M (default value) since the amount of memory per process is not specified.

#SBATCH --rsc p=6:t=4:c=4:m=20G
  • It allocate one process per four cores of the CPU and allocate one thread per physical core of the CPU and execute.
  • Please note the following process to allocate computing resources.
    • Our center manages memory in the form of tying memory to CPU cores, and in the case of system B, one CPU core must be allocated for every 4096 MB.
    • In this example, the amount of memory per process is specified as 20GB (m=20G), and from 20,000M/4,096M=4.88 cores, at least 5 cores must be allocated per process.
    • The larger value of the number of CPU cores specified by the --rsc option or the number of CPU cores required to allocate memory is used for the CPU cores allocated by the job scheduler. In this example, the number of CPU cores required to allocate memory exceeds the number of cores specified by the --rsc option, so the resources are allocated that the number of CPU cores used by one process is 5 cores. Please note that the process allocation is based on the values of c=4 and t=4 described in the --rsc option, so the remaining one core becomes an idle resource and is not actually used.

#SBATCH --rsc p=1:t=224:c=112:m=500G
  • Allocate 112 physical cores on one node from p=1,c=112 and start 2 threads on one physical core from t=224.
  • All memory available in one node (500 GB) is allocated from m=500G.
  • Please note that if the value of m is not specified, the available memory will be the default of 4096 MB.