Skip to content

Scratch System User Guide

What is Scratch and Why Does It Exist?

When you submit a job on the cluster, your files live in your home directory on NFS (a network filesystem). NFS is slow — typically 25–100 MB/s. The Lustre scratch filesystem is a high-performance parallel storage system capable of 500–900 MB/s.

The scratch system automatically:

  1. Copies your files from home → Lustre scratch before your job starts
  2. Your job runs on the fast Lustre filesystem
  3. After the job finishes, results are copied back to your home directory

Best Practice

You don't need to do anything manually — copying happens automatically in the background. Just add source .activate_scratch to your job script.

Step-by-Step: What Happens Under the Hood

1. PROLOG (before your job starts)

When SLURM accepts your job, it runs a prolog script before your code executes.

The prolog:

  • Creates your scratch directory: /lustre/scratch/<username>/job_<JOBID>/
  • Sets the Lustre stripe layout (optimizes parallel I/O)
  • Launches copying of your $SUBMIT_DIR to scratch — runs in the background on data01
  • Creates a .activate_scratch file in your submit directory
  • Exits quickly — copying continues asynchronously

What is SUBMIT_DIR?

SUBMIT_DIR is the directory from which you ran sbatch job.sh. All files in this directory are copied to scratch.

2. Waiting for the Copy to Finish

Copying runs in the background. Until it completes, your job waits for the .copy_ready marker file. You can monitor progress with:

tail -F /path/to/submit/dir/.prolog_progress_<JOBID>.log

3. JOB Runs

Once copying is complete, your job activates. Sourcing .activate_scratch sets the following environment variables:

  • $SCRATCH_DIR → /lustre/scratch/<username>/job_<JOBID>/
  • $TMPDIR → scratch/tmp
  • $RESULTS_DIR → scratch/results
  • Working directory is changed to scratch

Important

Your job should read and write to $SCRATCH_DIR, not to your home directory. Writing to NFS during a job defeats the purpose of scratch and will be significantly slower.

4. EPILOG (after your job finishes)

After your job ends, SLURM runs the epilog script. It:

  • Copies only new or modified files from scratch back to your submit directory — into results_job_<JOBID>/
  • Your original files in home are never overwritten
  • The scratch directory is deleted

You can monitor epilog progress with:

tail -F /path/to/submit/dir/.epilog_progress_<JOBID>.log

Where Are My Results?

After your job finishes, results appear in:

<your submit directory>/results_job_<JOBID>/

Example Use Case

You run sbatch from /home/ke001/experiment/. After the job completes, your results are at: 

/home/ke001/experiment/results_job_12345/
Input files that existed before the job are not copied — only files your job created or modified.

How to Write a Job Script with Scratch

#!/bin/bash
#SBATCH --job-name=my_job
#SBATCH --nodes=1
#SBATCH --ntasks=8
#SBATCH --time=02:00:00

# Activate scratch environment — sets $SCRATCH_DIR and changes working directory
source .activate_scratch

# You are now on Lustre scratch — fast storage
echo "Working in: $(pwd)"
echo "Scratch dir: $SCRATCH_DIR"

# Run your computation — read/write to $SCRATCH_DIR
./my_program --input $SCRATCH_DIR/input.dat --output $SCRATCH_DIR/output.dat

# No need to manually copy results back — the epilog does it automatically

Best Practice

Always use $SCRATCH_DIR, $TMPDIR, and $RESULTS_DIR variables in your scripts rather than hardcoding paths. This makes your scripts portable across different job IDs.

Special Files Reference

File Purpose
.activate_scratch Source this in your job script to activate the scratch environment
.prolog_progress_<JOBID>.log Live progress of data being copied to scratch
.epilog_progress_<JOBID>.log Live progress of results being copied back to home
.copy_ready Internal marker — when it exists, copying to scratch is complete
.no_scratch Create this file in your submit dir to skip scratch entirely

Skipping Scratch (.no_scratch mode)

If your job doesn't need fast I/O, or your dataset is too large to copy, you can opt out:

touch /path/to/submit/dir/.no_scratch

Important

In .no_scratch mode your job runs directly from NFS. I/O will be significantly slower. Only use this if your job is not I/O intensive or if data size makes copying impractical.

What Happens When You Cancel a Job (scancel)

Situation What happens
Job cancelled before copy finishes Copy is stopped, scratch is deleted, no results are synced back
Job cancelled after copy finishes Epilog runs normally — results appear in results_job_<JOBID>/

Important

If you cancel a job before copying completes, no results will be saved. If you need to stop a long-running job but want to keep partial results, wait for .copy_ready to exist before cancelling.

Automatic Cleanup

Important

Scratch directories older than 7 days are automatically deleted. Do not rely on data persisting on scratch long-term — your results are saved to results_job_<JOBID>/ in your home directory after the job ends.

Directory Layout at a Glance

/lustre/scratch/<username>/job_<JOBID>/          ← fast scratch during the job
/lustre/scratch/<username>/job_<JOBID>/tmp/      ← temp files ($TMPDIR)
/lustre/scratch/<username>/job_<JOBID>/results/  ← write results here ($RESULTS_DIR)

<submit_dir>/results_job_<JOBID>/                ← results after job ends (in home)
<submit_dir>/.prolog_progress_<JOBID>.log        ← copy-to-scratch progress log
<submit_dir>/.epilog_progress_<JOBID>.log        ← copy-back progress log

Subdirectories on Scratch

The following subdirectories are created automatically on scratch: 

job_<JOBID>/tmp/      ← temporary files
job_<JOBID>/logs/     ← log files
job_<JOBID>/output/   ← output files
job_<JOBID>/results/  ← final results ($RESULTS_DIR)