Skip to main content

Slurm Accounting and Resource Tracking

How Slurm tracks resource consumption through account hierarchies, TRES billing, and resource limits — sacctmgr, sreport, and the association model explained.

Why Accounting Matters

On a shared HPC cluster, resources are finite. Without accounting, a single user running large training jobs could starve everyone else. Slurm’s accounting subsystem solves this by tracking who uses what, enforcing per-user and per-group limits, and feeding usage data into the fair-share scheduler.

Accounting also enables capacity planning. By analyzing historical usage with sreport, administrators can justify hardware purchases, identify underutilized partitions, and set allocation targets that reflect actual demand.

The Association Model

Slurm organizes users into a tree: cluster → account → user. Every user must belong to an account, and accounts can be nested. This hierarchy is managed by sacctmgr, the accounting management command.

bash
# Create an account under the root
sacctmgr add account ml-group Description="Machine Learning Team"

# Create a child account
sacctmgr add account nlp-team parent=ml-group

# Add a user to an account
sacctmgr add user alice Account=ml-group

# View the full association tree
sacctmgr show associations format=Cluster,Account,User,Share,GrpTRES

Every row in this tree is called an association. An association ties a user to an account on a specific cluster and defines what resources they can consume. A user can belong to multiple accounts (e.g., a professor in both cs-dept and interdisciplinary-lab), but one association is marked as the default.

Parent-Child Inheritance

Resource limits cascade downward. If ml-group has a GrpTRES limit of 20 GPUs, all users under ml-group combined cannot exceed 20 GPUs — even if each user’s individual limit is set to 16.

Account Hierarchy Explorer

Visualize Slurm's association model: cluster → account → user hierarchy with cascading resource limits (GrpTRES) and usage tracking.

Association Tree

Node Details

hpc-clustercluster

Resource Limits (GrpTRES)

CPU Cores0 / 256
GPUs0 / 32
Memory (GB)0 / 2048

Children

ml-group
0/160 CPU · 0/20 GPU
cv-group
0/96 CPU · 0/12 GPU

Job Submission Simulation

Queued jobs (5):
1train-bert
alice · 32C 4G 256M
2finetune-vit
bob · 16C 2G 128M
3train-llm
alice · 48C 6G 384M
4detect-objects
carol · 32C 4G 256M
5scale-experiment
bob · 64C 8G 512M

Try a sacctmgr Command

Examples:
Step 1 of 7

Welcome

Explore how Slurm organizes users into accounts with cascading resource limits.

TRES: Trackable Resources

TRES (Trackable Resources) is Slurm’s unified system for metering different resource types. Instead of tracking CPUs, GPUs, and memory separately, everything is expressed in TRES units with configurable billing weights.

Billing Weights

In slurm.conf, administrators define how expensive each resource type is:

text
TRESBillingWeights="CPU=1.0,Mem=0.25G,GRES/gpu=16.0"

This says: 1 GPU costs as much as 16 CPUs. A job requesting 4 GPUs and 32 CPUs is billed as 4 × 16 + 32 × 1 = 96 billing units. This normalization lets fair-share treat CPU-heavy and GPU-heavy jobs equitably.

Why TRES Matters for Fair-Share

Without TRES weighting, a user running 100 small CPU jobs would accumulate the same “usage” as someone running 100 GPU jobs — despite consuming far fewer actual resources. TRES billing corrects this imbalance.

bash
# Check TRES billing for a completed job
sacct -j 12345 --format=JobID,TRESUsageInTot,TRESUsageOutTot

Resource Limits

Limits are the enforcement layer. They prevent any single user or group from consuming more than their allocation. Limits are set per-association via sacctmgr.

Group Limits (GrpTRES)

Cap the total resources consumed by an account and all its users combined:

bash
# ml-group can use at most 20 GPUs simultaneously
sacctmgr modify account ml-group set GrpTRES=gres/gpu=20,cpu=160

When a job would push the group total over GrpTRES, the job stays PENDING with reason AssocGrpCpuLimit or AssocGrpGRES.

Per-Job Limits (MaxTRES)

Cap what a single job can request:

bash
# No single job from alice can request more than 8 GPUs
sacctmgr modify user alice set MaxTRESPerJob=gres/gpu=8,cpu=64

Wall Time Limits

bash
# ml-group users cannot run jobs longer than 3 days
sacctmgr modify account ml-group set MaxWall=3-00:00:00

# Total accumulated wall time for the group
sacctmgr modify account ml-group set GrpWall=30-00:00:00

Limits Are Hard Caps

Unlike fair-share (which adjusts priority softly), GrpTRES and MaxTRES are hard limits. GrpTRES caps aggregate concurrent use: a pending job can start after other jobs release enough of the account's resources. For example, an 8-GPU job can fit a 20-GPU group cap once existing use falls from 16 to 12. An individual request larger than the fixed cap needs a request or policy change; waiting alone cannot make it fit. Check sacctmgr show associations when jobs remain pending with an AssocGrp reason.

Usage Reporting

sreport — Cluster and Account Utilization

The primary tool for understanding who used what over time:

bash
# Cluster utilization for the last month
sreport cluster utilization Start=2026-02-15 End=2026-03-15

# Top 10 users by GPU hours
sreport user TopUsage Start=2026-02-15 End=2026-03-15 \
  TopCount=10 -t Hours --tres=gres/gpu

# Account-level usage breakdown
sreport cluster AccountUtilizationByUser Start=2026-02-15 \
  End=2026-03-15 Accounts=ml-group

sacct — Job-Level Detail

While sreport gives aggregate views, sacct drills into individual jobs:

bash
# See TRES billing for recent jobs
sacct -u $USER --format=JobID,JobName,TRESUsageInTot,Elapsed,State \
  --starttime=2026-03-01

sshare — Fair-Share Impact

Usage feeds directly into fair-share scheduling. Heavy consumers get lower priority:

bash
sshare -u $USER -l
# Account  User   RawShares  NormShares  RawUsage  NormUsage  EffectvUsage  FairShare
# ml-group alice  100        0.10        45231     0.15       0.12          0.667

A FairShare value near 1.0 means you’ve used less than your share. Near 0.0 means you’ve consumed well beyond your allocation and your jobs will be deprioritized.

Common Pitfalls

1. Not Checking Association Limits

Your job is PENDING with reason AssocGrpGRES but you have free GPUs on the cluster. The problem: your account’s GrpTRES limit is exhausted. Check with sacctmgr show associations where user=$USER format=Account,GrpTRES.

2. Ignoring TRES Billing Weights

You submit 10 GPU jobs and wonder why your fair-share dropped much faster than a colleague who submitted 50 CPU jobs. Each GPU might be billed at 16x a CPU. Check sacctmgr show config | grep TRESBillingWeights.

3. Default Account Confusion

If you belong to multiple accounts, sbatch uses your default account unless you specify --account=. Running jobs under the wrong account skews fair-share for both groups.

4. Not Using sreport for Capacity Planning

Many teams request more allocation without data. sreport shows actual utilization — if your group only uses 40% of its GPU allocation, requesting more won’t help.

Further Reading

GPU & High-Performance Computing
Slurm Resource Management and Job Priority

How Slurm decides which jobs run first — priority factors, fair-share scheduling, backfill, and monitoring commands (squeue, sinfo, sacct).

GPU & High-Performance Computing
Slurm Backfill Scheduling: How Small Jobs Fill the Gaps

How sched/backfill works — the algorithm that lets small jobs run in gaps while large jobs wait, why accurate time limits matter, and the key tuning parameters (bf_interval, bf_window, bf_max_job_test).

GPU & High-Performance Computing
Slurm Fundamentals: Job Scheduling on HPC Clusters

Complete guide to Slurm — architecture, core commands, job lifecycle, job scripts, array jobs, dependencies, monitoring with squeue/sacct, and troubleshooting failed jobs on HPC clusters.

GPU & High-Performance Computing
Slurm GPU Allocation for Distributed Training

Complete guide to GPU allocation on Slurm — --gres flags, CUDA_VISIBLE_DEVICES remapping, GPU topology and NVLink binding, MIG partitioning, production job scripts, and debugging common GPU errors.

GPU & High-Performance Computing
HPC Performance Optimization: Scaling, Profiling, and Tuning

Amdahl and Gustafson ceilings, strong vs weak scaling, roofline bounds, and hiding all-reduce behind compute — the levers that decide whether more GPUs actually buy science.

GPU & High-Performance Computing
MPI Fundamentals: Message Passing for Distributed Computing

Complete MPI guide — point-to-point and collective communication with real C and mpi4py code, deadlock simulation, analytical performance modeling, communicator splitting, and debugging on HPC clusters.

If you found this explanation helpful, consider sharing it with others.

Mastodon