Stage a real EEG recording onto job-local storage

Estimated reading time:4 minutes

Use a small EEGDash recording to exercise the same stage-in workflow used on a cluster: persistent download, copy to local storage, then offline read. The demonstration downloads about 5.6 MB and copies actual BIDS files.

Prerequisites: a writable persistent cache, enough local scratch space, and EEGDash installed on the compute node. No Slurm allocation is needed to run the demonstration; a workstation temporary directory provides the same lifetime for the copied files. The recorded task is left/right imagery from nm000135, subject 1, session 0train, run 0. This page stages inputs, not trained model checkpoints.

1. Populate persistent storage on a network-enabled node

import os
from pathlib import Path

from eegdash import EEGDashDataset
from eegdash.paths import get_default_cache_dir

cache_dir = get_default_cache_dir()
query = dict(dataset="nm000135", subject="1", session="0train", run="0", task="imagery")

import shutil
import tempfile

Download before entering the temporary-directory context so acquisition failure cannot be mistaken for a compute-node problem. The source remains on persistent storage after the local copy is removed.

persistent = EEGDashDataset(cache_dir=cache_dir, **query, n_jobs=1)
persistent.download_all(n_jobs=1)
source = cache_dir / "nm000135"

2. Stage into a private directory on job-local storage

SLURM_TMPDIR identifies local scratch on some clusters. TemporaryDirectory makes this small demonstration runnable on a workstation as well; its lifecycle represents the job. The persistent source remains available. The dataset directory is copied with its relative BIDS structure, preserving channels and events beside the signals. This small example copies the whole cached nm000135 subtree: if earlier runs cached additional sessions, those files are copied too. Use a dedicated staging cache or select the required BIDS dependencies explicitly when that subtree no longer fits scratch.

A private temporary directory avoids two jobs overwriting the same staged files. It does not benchmark filesystem throughput or synchronize concurrent downloads into the persistent source; complete acquisition first.

scratch = os.environ.get("SLURM_TMPDIR")
with tempfile.TemporaryDirectory(prefix="eegdash-stage-", dir=scratch) as job_dir:
    local_cache = Path(job_dir)
    shutil.copytree(source, local_cache / "nm000135")
    staged = EEGDashDataset(cache_dir=local_cache, **query, download=False, n_jobs=1)
    raw = staged.datasets[0].raw
    print("Staged recording:", raw)
    print("Local cache:", local_cache)
    # Execute analysis here and return derived outputs to persistent storage.

Staged recording: <RawBDF | sub-1_ses-0train_task-imagery_run-0_eeg.bdf, 3 x 605000 (2420.0 s), ~7 KiB, data not loaded>
Local cache: /tmp/eegdash-stage-py1pwsmx

3. Apply the same pattern in a batch job

Pre-stage only the selected dataset before workers start. Point each worker’s EEGDASH_CACHE_DIR at the staged root and pass download=False. Save derived results back to persistent storage before the scheduler removes job scratch. Storage performance varies by cluster; measure it instead of assuming a particular speedup from the path name.

# 4. Check the storage boundary before scaling up
# -----------------------------------------------
# Opening verifies local discovery and the reader, not file integrity. Use
# checksums when transferring files between hosts.
# The displayed local path is valid only
# inside the with block. Put feature extraction or training there and copy its
# outputs to persistent storage before leaving the block, even if the shell
# job continues afterward.
#
# For a larger job, measure stage-in time separately from model time, include
# both source and destination space in your allocation, and record the exact
# query with the outputs. The Slurm EO/EC tutorial supplies a real training
# workload after this storage boundary has been verified.