SLURM GPU Cluster Job Scheduling & Failure Recovery

Requeues multi-node training on failure, and resumes from the last checkpoint with no one watching.

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

SLURM queues jobs, assigns whole nodes and starts your processes. Long training runs hit time limits and bad nodes, so a good job plans for interruption: save on a warning signal, requeue itself, and resume automatically.

Why it matters

Requeues multi-node training on failure, and resumes from the last checkpoint with no one watching.

1.2Key concepts

One idea per card, in plain words.

01Partition and QoS

Partitions group nodes; QoS sets limits and priority.

02gres

Requests generic resources such as GPUs: --gres=gpu:8 per node.

03--signal

Sends your script a warning before the time limit, so it can save.

04--requeue

Lets the job return to the queue after a failure or preemption.

05Health check

A program SLURM runs on each node. A failing GPU drains the node before it wastes jobs.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

A 4-node training job dies at SLURM's 24-hour limit and sometimes on a bad GPU. Write a job that resumes on its own, and explain each step.

Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Step by step

The mechanism, in the order you would explain it.

  1. Request resources

    Ask for nodes, GPUs per node and CPUs. One task per node, and torchrun starts the rest.

  2. Find the rendezvous

    Use the first hostname as the master address.

  3. Catch the warning

    --signal=B:USR1@300 tells the batch script five minutes before the limit.

  4. Save and requeue

    The trap requeues the job and asks training to stop, which saves on SIGTERM.

  5. Resume automatically

    train.py finds the newest complete checkpoint and continues from there.

  6. Drain bad nodes

    A node health check marks failing GPUs, so the requeued job avoids them.

3.2Code

Working code for the core of it.

A training job that survives preemption

Long jobs outlive their time limit. This one saves and goes back in the queue.

train.sbatchShellCopy
1#!/bin/bash2#SBATCH --job-name=train-70b3#SBATCH --nodes=44#SBATCH --ntasks-per-node=15#SBATCH --gres=gpu:86#SBATCH --cpus-per-task=647#SBATCH --time=24:00:008#SBATCH --requeue9#SBATCH --signal=B:USR1@300        # warn this script 5 minutes before the time limit10#SBATCH --output=logs/%x-%j.out11 12# On the warning: stop cleanly (training saves on SIGTERM), then go back in the queue.13trap 'scontrol requeue "$SLURM_JOB_ID"; kill -TERM "$TRAIN_PID"' USR114 15export MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1)16export MASTER_PORT=2950017 18srun torchrun --nnodes="$SLURM_NNODES" --nproc-per-node=8 \19  --rdzv-backend=c10d --rdzv-endpoint="$MASTER_ADDR:$MASTER_PORT" \20  train.py --resume-from-latest &21TRAIN_PID=$!22wait "$TRAIN_PID"
22 linesUTF-8Shell
--signal=B:USR1@300
Warns the script 5 minutes before the time limit.
trap 'scontrol requeue ...'
On the warning, it requeues, then stops training cleanly.
--resume-from-latest
The next run continues from the last complete checkpoint.

Syntax-checked only.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why the B: in --signal?
It sends the signal to the batch shell, where your trap runs. Without it the signal goes to job steps.
What if a node dies with no warning?
No signal arrives, so you rely on your last periodic checkpoint. That is why checkpoints must be regular and atomic.
SLURM or Kubernetes?
SLURM suits large multi-node training with gang scheduling. Kubernetes suits serving and mixed workloads. Many teams run both.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Partial checkpoints
A kill during a save must never look complete. Write a commit marker last.
Counterpoint
For small teams, a managed training service costs more per hour and saves a cluster administrator's salary.