Pretraining REVE#
This page pretrains the REVE foundation model from scratch on EEGDash Corpus v0, in three
steps: download the corpus, install, pretrain. The two scripts it uses, download.sh
and pretrain.sh, come from the code repository eegdash/eegdash-corpus and are shown in full below. They run
REVE’s own training code with the settings of the paper, whose runs used 16 GPUs on 4
machines; the same commands work on one machine with fewer GPUs, only slower. The paper
(arXiv:2611.06356) and the repository’s README
hold the details this page leaves out.
Before you start. A Linux machine with NVIDIA GPUs and Python 3.11, about 4 TB of
free disk (the store is 3.60 TB), and a Hugging Face account that accepted the terms
of the store (2. Download). Where there is no python3.11 command, any Python
3.11 does in its place in the two venv lines below, a conda environment’s python
for instance.
1. Download the corpus#
The code repository first, then the newest Hugging Face client in a small Python environment of its own, then the download. Everything on this page goes side by side into the folder you start in, so pick one outside a small home directory:
git clone https://github.com/eegdash/eegdash-corpus.git && cd eegdash-corpus
python3.11 -m venv ../hf-env && source ../hf-env/bin/activate
pip install -U huggingface_hub hf_xet
hf auth login # paste a token of your Hugging Face account
bash download.sh /data
deactivate
This fetches the store at tag v1.0 into /data/eegdash-corpus-v0 and prints the
path the last step takes. One bar counts the finished files. The download has its own
environment because the training environment of the next step pins an older Hugging Face
client, which spends one request per file, and a free account gets 1,000 requests per
five minutes: with that client the script still finishes, pausing now and then.
The download runs for hours, so start it where it survives a logout (tmux,
screen or a job); run again after an interruption, it goes on where it stopped.
VERIFY=1 JOBS=16 bash download.sh /data also checks every file against the published
SHA-256 list and ends with checksums OK. On an HPC cluster the download took
about 5.5 hours (near 200 MB/s) and the check about 3.5 hours.
download.sh:
#!/bin/bash
# Download eegdash-corpus-v0, the converted Zarr store (3.6 TB, 44,801 files), from Hugging Face.
#
# bash download.sh /data # -> /data/eegdash-corpus-v0/eegdash-corpus-v0.zarr
# VERIFY=1 JOBS=16 bash download.sh /data # also check every file against the published SHA-256 list
#
# Needs a Hugging Face account that accepted the dataset's terms, a token on this machine (`hf auth login`) and the
# newest Hugging Face client, in an environment of its own (README, Quick start, step 1). The training environment
# pins an older client that spends a request per file, and a free account gets 1,000 per five minutes: with it the
# script still finishes, pausing whenever the Hub refuses. After an interruption, run the same line again.
set -euo pipefail
OUT=${1:?usage: bash download.sh <a directory with 4 TB free>}
DEST=$OUT/eegdash-corpus-v0
hf auth whoami > /dev/null || { echo "not logged in: run 'hf auth login' and accept the terms on the dataset's page" >&2; exit 1; }
python - "$DEST" << 'EOF'
import os, sys, time, traceback
from pathlib import Path
from huggingface_hub import snapshot_download
from huggingface_hub.utils import disable_progress_bars
dest = Path(sys.argv[1])
listed = dest / "eegdash-corpus-v0-SHA256SUMS" # names every file of the store
for per_file in ("huggingface_hub.xet_get", "huggingface_hub.http_get"):
disable_progress_bars(per_file) # leaves one bar that counts the finished files
while True:
try:
snapshot_download("EEGDash/eegdash-corpus-v0", repo_type="dataset", revision="v1.0", local_dir=dest)
if listed.exists() and all((dest / line.split()[1]).exists() for line in listed.open()):
break # all there (a refused request can also return early)
except Exception as error:
if "429 Client Error" not in "".join(traceback.format_exception(error)):
raise # anything but the rate limit: stop
print("paused by the Hub's rate limit: finished files are kept, going on in a minute", flush=True)
time.sleep(float(os.environ.get("WAIT", 60)))
EOF
if [ "${VERIFY:-0}" = 1 ]; then # hashes every file, JOBS at a time: hours on the full store
parts=$(mktemp -d)
split -l 500 "$DEST/eegdash-corpus-v0-SHA256SUMS" "$parts/part."
(cd "$DEST" && ls "$parts"/part.* | xargs -P "${JOBS:-8}" -n 1 sha256sum --quiet -c)
rm -r "$parts"
echo "checksums OK"
fi
echo "DATA=$DEST/eegdash-corpus-v0.zarr"
2. Install#
A second environment with the packages training needs, and REVE’s training code with the repository’s patches, which let it read the Zarr store:
python3.11 -m venv ../reve-env && source ../reve-env/bin/activate
pip install -r requirements-pretraining.txt
git clone https://github.com/elouayas/reve_eeg.git ../reve_eeg && git -C ../reve_eeg checkout 06a7059
bash patches/apply_patches.sh ../reve_eeg
3. Pretrain REVE#
DATA=/data/eegdash-corpus-v0/eegdash-corpus-v0.zarr CKPT=/ckpt/reve-base-v0 bash pretrain.sh
On one machine this uses every GPU it has. The paper’s runs used 16 GPUs: 4 machines with 4 GPUs each. On several machines, run the same line once on each machine, with three more variables:
# on the first of 4 machines; the same line on the others with NODE_RANK=1, 2 and 3
NNODES=4 NODE_RANK=0 MAIN_HOST=node01 \
DATA=/data/eegdash-corpus-v0/eegdash-corpus-v0.zarr CKPT=/ckpt/reve-base-v0 bash pretrain.sh
NNODES: the number of machines, 4 here.NODE_RANK: the index of the machine the line runs on, 0 on the first, then 1, 2 and 3.MAIN_HOST: the name of the first machine, the one withNODE_RANK=0; the others connect to it.
The GPUs of each machine are counted as on one machine, so 4 machines with 4 GPUs give
16 training processes. Every machine needs the same number of GPUs and must see the same
DATA and CKPT folders; training starts once the line runs on all of them.
pretrain.sh:
#!/bin/bash
# Pretrain REVE-Base from scratch on eegdash-corpus-v0 with the paper's settings (17 epochs, peak learning rate
# 2.4e-4, global batch 300), with REVE's own trainer (reve_eeg) patched to read the Zarr store.
#
# DATA=/data/eegdash-corpus-v0/eegdash-corpus-v0.zarr CKPT=/ckpt/reve-base-v0 bash pretrain.sh
#
# One machine: every GPU it has is used. Several machines (the paper: 16 GPUs on 4 machines): run the same line once
# on each machine, with the number of machines, this machine's index and the first machine's name:
#
# NNODES=4 NODE_RANK=0 MAIN_HOST=node01 DATA=... CKPT=... bash pretrain.sh # on node01; NODE_RANK=1, 2, 3 on the others
#
# REVE's patched code is expected beside this repository (../reve_eeg, as the Quick start clones it); REVE_EEG names
# another place. Run again with the same CKPT, it continues after the last saved epoch. It writes $CKPT/run/encoder.pth (the
# pretrained encoder) and $CKPT/run/loss.csv. Every other setting (SEED, EPOCHS, ENCODER=small, WORKERS,
# MIXED_PRECISION, ...) is train.sh's, which this script calls: its header lists them.
set -euo pipefail
HERE=$(cd "$(dirname "$0")" && pwd)
: "${DATA:?set DATA to the eegdash-corpus-v0.zarr store (download.sh prints it)}"
: "${CKPT:?set CKPT to a directory for the checkpoints}"
export DATA CKPT SEED=${SEED:-2} REVE_EEG=${REVE_EEG:-$(dirname "$HERE")/reve_eeg} # REVE's code: beside this repository unless REVE_EEG says otherwise
export RUN=${RUN:-reve-base-v0-seed$SEED}
grep -q "PATCH 8" "$REVE_EEG/src/utils/data_loading.py" 2> /dev/null || {
echo "no patched reve_eeg at $REVE_EEG: clone it at 06a7059 and run patches/apply_patches.sh (README, Quick start)" >&2; exit 1; }
GPUS=${GPUS:-$(nvidia-smi -L | wc -l)} # the GPUs of this machine
export NNODES=${NNODES:-1}
export NPROC=$((NNODES * GPUS)) # processes over all machines, one per GPU
if [ "$NNODES" -gt 1 ]; then # where the machines find each other
: "${MAIN_HOST:?several machines: set MAIN_HOST to the name of the first machine}"
: "${NODE_RANK:?several machines: set NODE_RANK to the index of this machine, 0 on the first}"
export ACCELERATE_ARGS="--machine_rank $NODE_RANK --main_process_ip $MAIN_HOST --main_process_port 29500"
fi
if [ -d "$CKPT/states/$RUN/last" ]; then export RESUME=True; fi
bash "$HERE/train.sh"
4. What you get#
The pretrained encoder, $CKPT/run/encoder.pth, and the loss of every epoch,
$CKPT/run/loss.csv. The paper trained REVE-Base twice this way (seeds 2 and 3), 17
epochs each on 16 GPUs across 4 machines: 23.5 hours on A40 GPUs and 29.7 hours on V100
GPUs.
The two encoders, scored on seven public OpenEEGBench tasks, against the released REVE-Base (chance-adjusted balanced accuracy, 0 = chance; mean of the two runs):
Linear probe |
PhysioNet-MI |
BCIC-IV-2a |
BCIC-2020-3 |
FACED |
Mumtaz |
Zyma |
CHB-MIT |
mean |
|---|---|---|---|---|---|---|---|---|
released REVE-Base |
0.377 |
0.191 |
0.088 |
0.167 |
0.681 |
0.295 |
0.529 |
0.333 |
pretrained on the corpus |
0.373 |
0.230 |
0.095 |
0.159 |
0.694 |
0.340 |
0.423 |
0.331 |
Full fine-tuning |
PhysioNet-MI |
BCIC-IV-2a |
BCIC-2020-3 |
FACED |
Mumtaz |
Zyma |
CHB-MIT |
mean |
|---|---|---|---|---|---|---|---|---|
released REVE-Base |
0.436 |
0.352 |
0.041 |
0.183 |
0.675 |
0.323 |
0.264 |
0.325 |
pretrained on the corpus |
0.379 |
0.296 |
0.041 |
0.160 |
0.667 |
0.226 |
0.144 |
0.273 |
The released model’s scores are the OpenEEGBench leaderboard’s; CHB-MIT, which the leaderboard does not score, is the released weights run through the same scoring.
To use the encoder, convert it to braindecode’s format with scripts/export_encoder.py
from the code repository, then fine-tune it as in
the REVE fine-tuning tutorial.
5. Settings of the scripts#
Every setting is an environment variable written before the command, as DATA and
CKPT are above; download.sh also takes one argument, the target directory.
5.1 download.sh#
bash download.sh <directory>
setting |
default |
meaning |
|---|---|---|
|
required |
The one argument: a directory with 4 TB free. The store lands in
|
|
|
|
|
|
With |
|
|
Seconds to wait before going on when the Hub’s rate limit pauses the download. |
5.2 pretrain.sh#
DATA=<store> CKPT=<directory> bash pretrain.sh
setting |
default |
meaning |
|---|---|---|
|
required |
The store, |
|
required |
The directory the run writes to. Run again with the same one, the run continues after its last saved epoch. |
|
|
Seeds the weights and the batch order. The paper’s two runs used 2 and 3. |
|
|
The run’s name: it names the saved states under |
|
|
The patched checkout of REVE’s code, by default beside this repository. |
|
all of the machine’s |
The number of GPUs to use on each machine. |
|
|
The number of machines. |
|
required when |
The index of the machine the line runs on, 0 on the first. |
|
required when |
The name of the first machine; the others connect to it. |
5.3 train.sh#
DATA=<store> RUN=<name> CKPT=<directory> REVE_EEG=<checkout> SEED=<n> bash train.sh
pretrain.sh calls train.sh, the script that launches REVE’s trainer, and passes
these settings on; they can be set on the pretrain.sh line too. Called by itself,
train.sh needs DATA, RUN, CKPT, REVE_EEG and SEED, and takes
NNODES (default 1) and NPROC, the number of processes over all machines
(default 1).
setting |
default |
meaning |
|---|---|---|
|
|
The number of epochs; 17 is the paper’s. |
|
|
The model size, one of the encoder configurations of REVE’s code: |
|
|
The peak learning rate. |
|
|
REVE’s batch-size setting: a batch holds 16 × |
|
|
Data-loading workers per GPU. |
|
|
|
|
none |
Dataset ids left out of training, separated by commas
( |
|
|
Saves a snapshot of the encoder, |
|
|
|
|
none |
Extra flags for |
A run writes under $CKPT: run/encoder.pth (the pretrained encoder, at the end),
run/encoder_epoch_<k>.pth (the snapshots), run/loss.csv (one row per epoch) and
states/$RUN/ (a state per epoch, to continue from).