Note
Go to the end to download the full example code or to run this example in your browser via Binder.
Track 1: retrieve actual viewed images from recorded EEG#
Load THINGS-EEG2 subject 08, session 02, task-test from nm000232. This is one whole recording (hundreds of MB) plus 60 original stimulus images. Observed tot_img_number values in the BIDS event sidecar identify the images; the release’s stimuli.tsv maps those IDs to the actual JPG files.
A frozen DINOv2-small encoder supplies 384-dimensional image targets. Ridge maps EEG to these real image features. Split image identities 60/20/20 for training, validation and testing. Each retrieval gallery contains only that split’s unseen images. The default uses nonofficial warm-up targets; EEGDASH_DINO_MODEL=giant opts into the costly official target encoder. The source calls this recording “test”, but our within-recording image split is an instructional experiment, not the official challenge train/test split.
Source: nemarDatasets/nm000232
Before you start#
Use EEGDash, NeuralSet, Transformers, TorchVision, MNE, NumPy, pandas,
scikit-learn, Pillow and Matplotlib. The BDF is about 272 MB; first use also
fetches 60 JPGs and a pretrained checkpoint (about 88 MB for DINOv2-small,
about 4.5 GB for giant). CPU execution uses one image at a time. NeuralSet
caches extracted targets, and Hugging Face retains the model checkpoint.
EEGDASH_CACHE_DIR retains both the signal and things_test_images.
Network failures are acquisition errors and are not replaced with substitute
targets.
The official track
retrieves unseen images using frozen DINOv2 features. The NeuralBench guide
specifies giant features at relative depth 0.6667, mean token pooling and
imsize=518, scored across the complete held-out gallery and then averaged
across participants. Its within-batch validation metric is a different task.
The opt-in giant encoder uses the guide’s NeuralSet extraction settings. The default small model reduces download and CPU cost. Neither mode reproduces its contrastive EEG training, source-defined splits or hidden Alljoined cohort. This one-session ridge warm-up reports its own held-out gallery size alongside its measured score.
import os
from pathlib import Path
from urllib.request import urlopen
import matplotlib.pyplot as plt
import mne
import numpy as np
import pandas as pd
import torch
from neuralset.events.etypes import Image as ImageEvent
from neuralset.extractors import HuggingFaceImage
from PIL import Image
from sklearn.linear_model import Ridge
from sklearn.metrics import top_k_accuracy_score
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from eegdash.paths import get_default_cache_dir
from eegdash import EEGDash, EEGDashDataset
Choose one converted recording and its observed image IDs#
The catalogue also contains original BrainVision source files. Selecting
a path beginning with sub- chooses the converted BIDS BDF and its event
sidecar rather than loading both representations of the same run.
The exact path is printed so the acquisition remains reviewable.
tot_img_number in that sidecar records which stimulus was presented.
Normal image trials with IDs 1–60 are kept, excluding target/catch trials.
Subtracting one only changes these observed IDs into zero-based array indices.
Numeric trigger order is never guessed to be an image identity.
records = EEGDash().find(
{"dataset": "nm000232", "subject": "08", "session": "02", "task": "test"}
)
records = [record for record in records if record["bids_relpath"].startswith("sub-")]
cache = get_default_cache_dir()
dataset = EEGDashDataset(records=records, cache_dir=cache)
print(dataset.description.to_string(index=False))
print("Exact signal:", records[0]["bids_relpath"])
raw = dataset.datasets[0].raw.pick("eeg")
# Read the sidecar acquired together with this exact EEG recording.
event_path = Path(raw.filenames[0]).with_name(
Path(raw.filenames[0]).name.replace("_eeg.bdf", "_events.tsv")
)
trials = pd.read_csv(event_path, sep="\t")
trials = trials[
trials.trial_type.eq("image") & trials.tot_img_number.between(1, 60)
].copy()
labels = trials.tot_img_number.to_numpy(dtype=int) - 1
# Pin the metadata and JPG revision together; no inferred trigger-to-image mapping.
subject session task age sex n_sessions run gender
08 02 test 25.0 F 4.0 None None
Exact signal: sub-08/ses-02/eeg/sub-08_ses-02_task-test_eeg.bdf
[10/15/26 13:13:48] INFO HTTP Request: GET _client.py:1025
https://data.nemar.org/nm000232/
"HTTP/1.1 200 OK"
[10/15/26 13:13:49] INFO HTTP Request: GET _client.py:1025
https://data.nemar.org/nm000232/v1.
1.1/manifest.json "HTTP/1.1 200 OK"
INFO HTTP Request: GET _client.py:1025
https://data.nemar.org/nm000232/v1.
1.1/sub-08/ses-02/eeg/sub-08_ses-02
_task-test_channels.tsv "HTTP/1.1
200 OK"
INFO HTTP Request: GET _client.py:1025
https://data.nemar.org/nm000232/v1.
1.1/sub-08/ses-02/eeg/sub-08_ses-02
_task-test_events.tsv "HTTP/1.1 200
OK"
[10/15/26 13:13:50] INFO HTTP Request: GET _client.py:1025
https://data.nemar.org/nm000232/v1.
1.1/sub-08/ses-02/eeg/sub-08_ses-02
_task-test_events.json "HTTP/1.1
500 Internal Server Error"
[10/15/26 13:13:52] INFO HTTP Request: GET _client.py:1025
https://data.nemar.org/nm000232/v1.
1.1/sub-08/ses-02/eeg/sub-08_ses-02
_task-test_events.json "HTTP/1.1
200 OK"
INFO HTTP Request: GET _client.py:1025
https://data.nemar.org/nm000232/v1.
1.1/sub-08/ses-02/eeg/sub-08_ses-02
_task-test_eeg.json "HTTP/1.1 200
OK"
Downloading sub-08_ses-02_task-test_eeg.bdf: 0%| | 0.00/259M [00:00<?, ?B/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 0%| | 0.00/259M [00:00<?, ?B/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 0%| | 1.00M/259M [00:00<03:01, 1.49MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 1%| | 3.00M/259M [00:00<01:16, 3.53MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 3%|▎ | 9.00M/259M [00:01<00:28, 9.13MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 9%|▉ | 23.0M/259M [00:01<00:11, 20.9MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 13%|█▎ | 34.0M/259M [00:01<00:09, 26.1MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 19%|█▉ | 49.0M/259M [00:01<00:06, 33.0MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 24%|██▍ | 62.0M/259M [00:02<00:05, 36.7MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 29%|██▉ | 75.0M/259M [00:02<00:04, 40.4MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 34%|███▍ | 88.0M/259M [00:02<00:04, 43.5MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 39%|███▉ | 101M/259M [00:02<00:03, 44.6MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 45%|████▍ | 116M/259M [00:03<00:03, 47.3MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 47%|████▋ | 121M/259M [00:03<00:03, 44.9MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 52%|█████▏ | 134M/259M [00:03<00:02, 46.7MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 58%|█████▊ | 149M/259M [00:03<00:02, 49.0MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 61%|██████ | 158M/259M [00:03<00:02, 48.1MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 67%|██████▋ | 173M/259M [00:04<00:01, 51.2MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 71%|███████ | 184M/259M [00:04<00:01, 50.1MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 77%|███████▋ | 199M/259M [00:04<00:01, 51.7MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 82%|████████▏ | 213M/259M [00:04<00:00, 53.2MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 86%|████████▌ | 222M/259M [00:05<00:00, 51.9MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 90%|█████████ | 234M/259M [00:05<00:00, 52.4MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 96%|█████████▌| 248M/259M [00:05<00:00, 54.4MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 100%|█████████▉| 258M/259M [00:05<00:00, 53.7MB/s]
Downloading sub-08_ses-02_task-test_eeg.bdf: 100%|██████████| 259M/259M [00:05<00:00, 47.0MB/s]
[10/15/26 13:13:58] INFO HTTP Request: GET _client.py:1025
https://data.nemar.org/nm000232/v1.
1.1/dataset_description.json
"HTTP/1.1 200 OK"
INFO HTTP Request: GET _client.py:1025
https://data.nemar.org/nm000232/v1.
1.1/participants.tsv "HTTP/1.1 200
OK"
[10/15/26 13:13:59] INFO HTTP Request: GET _client.py:1025
https://data.nemar.org/nm000232/v1.
1.1/participants.json "HTTP/1.1 200
OK"
INFO HTTP Request: GET _client.py:1025
https://data.nemar.org/nm000232/v1.
1.1/README.md "HTTP/1.1 500
Internal Server Error"
[10/15/26 13:14:00] INFO HTTP Request: GET _client.py:1025
https://data.nemar.org/nm000232/v1.
1.1/README.md "HTTP/1.1 200 OK"
[10/15/26 13:14:01] INFO HTTP Request: GET _client.py:1025
https://data.nemar.org/nm000232/v1.
1.1/CHANGES "HTTP/1.1 200 OK"
INFO HTTP Request: GET _client.py:1025
https://data.nemar.org/nm000232/v1.
1.1/.bidsignore "HTTP/1.1 200 OK"
Build targets from the actual stimulus files#
The pinned stimulus manifest maps each observed ID to its original JPG. Downloaded bytes must decode as images before entering the cache. Image IDs come from the observed event sidecar, rather than a guessed trigger ordering. The frozen encoder sees only original images; all EEG-to-target fitting and hyperparameter selection happen after the image-identity split below.
NeuralSet performs the same relative-layer selection and token averaging in
either model mode. The small model yields 384 coordinates; giant yields 1536.
Its image processor can resize/crop after imsize: using the public extractor
preserves the guide’s actual processing path rather than hand-implementing a
superficially similar resize. Unit normalization gives cosine ranking a common
scale. These are measured pretrained activations, never random image targets.
revision = "5732fd1a9b9c873c94b25c7e7e70e550dc64faa7"
root = f"https://raw.githubusercontent.com/nemarDatasets/nm000232/{revision}/stimuli/"
manifest = pd.read_csv(root + "stimuli.tsv", sep="\t").set_index("stimulus_id")
image_cache = cache / "things_test_images" / revision
image_cache.mkdir(parents=True, exist_ok=True)
image_events = []
for identity in range(1, 61):
row = manifest.loc[f"stim-test{identity:03d}"]
destination = image_cache / Path(row.filename).name
if not destination.exists():
with urlopen(root + row.filename, timeout=60) as response:
payload = response.read()
# Decode before committing to cache: failed acquisitions cannot masquerade as images.
from io import BytesIO
with Image.open(BytesIO(payload)) as image:
image.verify()
destination.write_bytes(payload)
image_events.append(
ImageEvent(filepath=destination, start=0, duration=1, timeline="stimulus")
)
model_size = os.environ.get("EEGDASH_DINO_MODEL", "small")
model_revision = {
"small": "ed25f3a31f01632728cabb09d1542f84ab7b00566",
"giant": "611a9d42f2335e0f921f1e313ad3c1b7178d206d",
}[model_size]
torch.set_num_threads(2)
image_encoder = HuggingFaceImage(
model_name=f"facebook/dinov2-{model_size}",
pretrained=True,
layers=0.6667,
token_aggregation="mean",
imsize=518,
batch_size=1,
device="cpu",
hf_config={
"model_kwargs": {"revision": model_revision},
"processor_kwargs": {"revision": model_revision},
},
infra={"folder": cache / "track1_dinov2", "cluster": None},
)
image_encoder.prepare(image_events)
embeddings = np.stack(
[image_encoder.get_static(event).numpy() for event in image_events]
)
embeddings /= np.linalg.norm(embeddings, axis=1, keepdims=True)
# Limit windows to 180 ms: stimuli occur approximately every 200 ms. Earlier
# responses can still overlap because this is a rapid serial presentation task.
/home/runner/work/EEGDash/EEGDash/.venv/lib/python3.12/site-packages/neuralset/utils.py:86: UserWarning: The effect of "imsize"=518 might be cancelled by the HuggingFace processor.
warnings.warn(message)
Fetching 6 files: 0%| | 0/6 [00:00<?, ?it/s]
Fetching 6 files: 17%|█▋ | 1/6 [00:00<00:00, 6.62it/s]
Fetching 6 files: 83%|████████▎ | 5/6 [00:01<00:00, 4.17it/s]
Fetching 6 files: 100%|██████████| 6/6 [00:02<00:00, 2.96it/s]
0%| | 0/60 [00:00<?, ?it/s][10/15/26 13:14:14] INFO Computing 60 image latents image.py:162
Computing image embeddings: 0%| | 0/60 [00:00<?, ?it/s]Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
[10/15/26 13:14:15] WARNING Using a slow image processor as logging.py:328
`use_fast` is unset and a slow
processor was saved with this model.
`use_fast=True` will be the default
behavior in v4.52, even if the model
was saved with a slow processor.
This will result in minor
differences in outputs. You'll still
be able to use a slow processor with
`use_fast=False`.
It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
WARNING It looks like you are trying to logging.py:328
rescale already rescaled images. If
the input images have pixel values
between 0 and 1, set
`do_rescale=False` to avoid
rescaling them again.
2%|▏ | 1/60 [00:02<01:58, 2.02s/it]
Computing image embeddings: 2%|▏ | 1/60 [00:01<01:34, 1.61s/it]
3%|▎ | 2/60 [00:02<00:51, 1.12it/s]
Computing image embeddings: 3%|▎ | 2/60 [00:01<00:42, 1.37it/s]
5%|▌ | 3/60 [00:02<00:30, 1.86it/s]
Computing image embeddings: 5%|▌ | 3/60 [00:01<00:25, 2.24it/s]
7%|▋ | 4/60 [00:02<00:20, 2.71it/s]
Computing image embeddings: 7%|▋ | 4/60 [00:01<00:17, 3.18it/s]
8%|▊ | 5/60 [00:02<00:15, 3.60it/s]
Computing image embeddings: 8%|▊ | 5/60 [00:02<00:13, 4.13it/s]
10%|█ | 6/60 [00:02<00:11, 4.50it/s]
Computing image embeddings: 10%|█ | 6/60 [00:02<00:10, 5.03it/s]
12%|█▏ | 7/60 [00:02<00:09, 5.33it/s]
Computing image embeddings: 12%|█▏ | 7/60 [00:02<00:09, 5.82it/s]
13%|█▎ | 8/60 [00:02<00:08, 6.11it/s]
Computing image embeddings: 13%|█▎ | 8/60 [00:02<00:07, 6.53it/s]
15%|█▌ | 9/60 [00:02<00:07, 6.75it/s]
Computing image embeddings: 15%|█▌ | 9/60 [00:02<00:07, 7.11it/s]
17%|█▋ | 10/60 [00:03<00:06, 7.29it/s]
Computing image embeddings: 17%|█▋ | 10/60 [00:02<00:06, 7.57it/s]
18%|█▊ | 11/60 [00:03<00:06, 7.69it/s]
Computing image embeddings: 18%|█▊ | 11/60 [00:02<00:06, 7.90it/s]
20%|██ | 12/60 [00:03<00:06, 7.99it/s]
Computing image embeddings: 20%|██ | 12/60 [00:02<00:05, 8.15it/s]
22%|██▏ | 13/60 [00:03<00:05, 8.23it/s]
Computing image embeddings: 22%|██▏ | 13/60 [00:02<00:05, 8.35it/s]
23%|██▎ | 14/60 [00:03<00:05, 8.32it/s]
Computing image embeddings: 23%|██▎ | 14/60 [00:03<00:05, 8.40it/s]
25%|██▌ | 15/60 [00:03<00:05, 8.40it/s]
Computing image embeddings: 25%|██▌ | 15/60 [00:03<00:05, 8.46it/s]
27%|██▋ | 16/60 [00:03<00:05, 8.53it/s]
Computing image embeddings: 27%|██▋ | 16/60 [00:03<00:05, 8.57it/s]
28%|██▊ | 17/60 [00:03<00:05, 8.59it/s]
Computing image embeddings: 28%|██▊ | 17/60 [00:03<00:04, 8.62it/s]
30%|███ | 18/60 [00:03<00:04, 8.55it/s]
Computing image embeddings: 30%|███ | 18/60 [00:03<00:04, 8.57it/s]
32%|███▏ | 19/60 [00:04<00:04, 8.56it/s]
Computing image embeddings: 32%|███▏ | 19/60 [00:03<00:04, 8.58it/s]
33%|███▎ | 20/60 [00:04<00:04, 8.65it/s]
Computing image embeddings: 33%|███▎ | 20/60 [00:03<00:04, 8.66it/s]
35%|███▌ | 21/60 [00:04<00:04, 8.68it/s]
Computing image embeddings: 35%|███▌ | 21/60 [00:03<00:04, 8.69it/s]
37%|███▋ | 22/60 [00:04<00:04, 8.72it/s]
Computing image embeddings: 37%|███▋ | 22/60 [00:04<00:04, 8.72it/s]
38%|███▊ | 23/60 [00:04<00:04, 8.75it/s]
Computing image embeddings: 38%|███▊ | 23/60 [00:04<00:04, 8.75it/s]
40%|████ | 24/60 [00:04<00:04, 8.71it/s]
Computing image embeddings: 40%|████ | 24/60 [00:04<00:04, 8.71it/s]
42%|████▏ | 25/60 [00:04<00:04, 8.72it/s]
Computing image embeddings: 42%|████▏ | 25/60 [00:04<00:04, 8.72it/s]
43%|████▎ | 26/60 [00:04<00:03, 8.75it/s]
Computing image embeddings: 43%|████▎ | 26/60 [00:04<00:03, 8.76it/s]
45%|████▌ | 27/60 [00:04<00:03, 8.80it/s]
Computing image embeddings: 45%|████▌ | 27/60 [00:04<00:03, 8.80it/s]
47%|████▋ | 28/60 [00:05<00:03, 8.82it/s]
Computing image embeddings: 47%|████▋ | 28/60 [00:04<00:03, 8.82it/s]
48%|████▊ | 29/60 [00:05<00:03, 8.82it/s]
Computing image embeddings: 48%|████▊ | 29/60 [00:04<00:03, 8.82it/s]
50%|█████ | 30/60 [00:05<00:03, 8.81it/s]
Computing image embeddings: 50%|█████ | 30/60 [00:04<00:03, 8.81it/s]
52%|█████▏ | 31/60 [00:05<00:03, 8.82it/s]
Computing image embeddings: 52%|█████▏ | 31/60 [00:05<00:03, 8.82it/s]
53%|█████▎ | 32/60 [00:05<00:03, 8.81it/s]
Computing image embeddings: 53%|█████▎ | 32/60 [00:05<00:03, 8.81it/s]
55%|█████▌ | 33/60 [00:05<00:03, 8.83it/s]
Computing image embeddings: 55%|█████▌ | 33/60 [00:05<00:03, 8.83it/s]
57%|█████▋ | 34/60 [00:05<00:02, 8.82it/s]
Computing image embeddings: 57%|█████▋ | 34/60 [00:05<00:02, 8.82it/s]
58%|█████▊ | 35/60 [00:05<00:02, 8.83it/s]
Computing image embeddings: 58%|█████▊ | 35/60 [00:05<00:02, 8.83it/s]
60%|██████ | 36/60 [00:06<00:02, 8.75it/s]
Computing image embeddings: 60%|██████ | 36/60 [00:05<00:02, 8.75it/s]
62%|██████▏ | 37/60 [00:06<00:02, 8.70it/s]
Computing image embeddings: 62%|██████▏ | 37/60 [00:05<00:02, 8.70it/s]
63%|██████▎ | 38/60 [00:06<00:02, 8.65it/s]
Computing image embeddings: 63%|██████▎ | 38/60 [00:05<00:02, 8.65it/s]
65%|██████▌ | 39/60 [00:06<00:02, 8.70it/s]
Computing image embeddings: 65%|██████▌ | 39/60 [00:05<00:02, 8.70it/s]
67%|██████▋ | 40/60 [00:06<00:02, 8.73it/s]
Computing image embeddings: 67%|██████▋ | 40/60 [00:06<00:02, 8.73it/s]
68%|██████▊ | 41/60 [00:06<00:02, 8.77it/s]
Computing image embeddings: 68%|██████▊ | 41/60 [00:06<00:02, 8.77it/s]
70%|███████ | 42/60 [00:06<00:02, 8.82it/s]
Computing image embeddings: 70%|███████ | 42/60 [00:06<00:02, 8.82it/s]
72%|███████▏ | 43/60 [00:06<00:01, 8.84it/s]
Computing image embeddings: 72%|███████▏ | 43/60 [00:06<00:01, 8.84it/s]
73%|███████▎ | 44/60 [00:06<00:01, 8.83it/s]
Computing image embeddings: 73%|███████▎ | 44/60 [00:06<00:01, 8.83it/s]
75%|███████▌ | 45/60 [00:07<00:01, 8.82it/s]
Computing image embeddings: 75%|███████▌ | 45/60 [00:06<00:01, 8.82it/s]
77%|███████▋ | 46/60 [00:07<00:01, 8.78it/s]
Computing image embeddings: 77%|███████▋ | 46/60 [00:06<00:01, 8.78it/s]
78%|███████▊ | 47/60 [00:07<00:01, 8.73it/s]
Computing image embeddings: 78%|███████▊ | 47/60 [00:06<00:01, 8.73it/s]
80%|████████ | 48/60 [00:07<00:01, 8.76it/s]
Computing image embeddings: 80%|████████ | 48/60 [00:06<00:01, 8.76it/s]
82%|████████▏ | 49/60 [00:07<00:01, 8.84it/s]
Computing image embeddings: 82%|████████▏ | 49/60 [00:07<00:01, 8.84it/s]
83%|████████▎ | 50/60 [00:07<00:01, 8.88it/s]
Computing image embeddings: 83%|████████▎ | 50/60 [00:07<00:01, 8.88it/s]
85%|████████▌ | 51/60 [00:07<00:01, 8.81it/s]
Computing image embeddings: 85%|████████▌ | 51/60 [00:07<00:01, 8.81it/s]
87%|████████▋ | 52/60 [00:07<00:00, 8.78it/s]
Computing image embeddings: 87%|████████▋ | 52/60 [00:07<00:00, 8.78it/s]
88%|████████▊ | 53/60 [00:07<00:00, 8.81it/s]
Computing image embeddings: 88%|████████▊ | 53/60 [00:07<00:00, 8.81it/s]
90%|█████████ | 54/60 [00:08<00:00, 8.84it/s]
Computing image embeddings: 90%|█████████ | 54/60 [00:07<00:00, 8.84it/s]
92%|█████████▏| 55/60 [00:08<00:00, 8.73it/s]
Computing image embeddings: 92%|█████████▏| 55/60 [00:07<00:00, 8.73it/s]
93%|█████████▎| 56/60 [00:08<00:00, 8.73it/s]
Computing image embeddings: 93%|█████████▎| 56/60 [00:07<00:00, 8.73it/s]
95%|█████████▌| 57/60 [00:08<00:00, 8.75it/s]
Computing image embeddings: 95%|█████████▌| 57/60 [00:07<00:00, 8.75it/s]
97%|█████████▋| 58/60 [00:08<00:00, 8.69it/s]
Computing image embeddings: 97%|█████████▋| 58/60 [00:08<00:00, 8.70it/s]
98%|█████████▊| 59/60 [00:08<00:00, 8.71it/s]
Computing image embeddings: 98%|█████████▊| 59/60 [00:08<00:00, 8.71it/s]
100%|██████████| 60/60 [00:08<00:00, 8.73it/s]
100%|██████████| 60/60 [00:08<00:00, 6.86it/s]
Computing image embeddings: 100%|██████████| 60/60 [00:08<00:00, 8.73it/s]
Computing image embeddings: 100%|██████████| 60/60 [00:08<00:00, 7.20it/s]
Extract short visual-response windows#
Event sample indices come from the same BDF sidecar. Posterior channels O1, Oz and O2 keep the predictor small. Windows run from onset to 180 ms because images arrive at roughly 200 ms intervals. This limits overlap with the next presentation but cannot remove responses to preceding images in the rapid stream; it is not isolated-trial visual decoding.
MNE handles epoch rejection and resamples the retained waveforms to 100 Hz.
Indexing labels by epochs.selection preserves their correspondence if an
epoch is dropped. Flattening yields (retained trials, channels × samples)
features in volts. Feature scaling is deferred until training rather than
estimated from the full collection.
events = np.column_stack(
[
trials["sample"].to_numpy(dtype=int),
np.zeros(len(trials), dtype=int),
np.ones(len(trials), dtype=int),
]
)
epochs = mne.Epochs(
raw,
events,
event_id={"image": 1},
tmin=0,
tmax=0.18,
baseline=None,
picks=["O1", "Oz", "O2"],
preload=True,
reject_by_annotation=True,
).resample(100)
labels = labels[epochs.selection]
X = epochs.get_data().reshape(len(epochs), -1)
print("Actual EEG features:", X.shape, "actual image features:", embeddings.shape)
Not setting metadata
1214 matching events found
No baseline correction applied
0 projection items activated
Loading data for 1214 events and 181 original time points ...
0 bad epochs dropped
Actual EEG features: (1214, 54) actual image features: (60, 384)
Split repeated presentations by image identity#
An image can occur many times in the recording. A random trial split would let the decoder see the same image during training and evaluation, answering an easier repeated-stimulus question. The seeded permutation instead assigns complete image identities: 36 train, 12 validate, and 12 test when all 60 are retained. It randomizes only assignments, not signals or labels.
These are within-participant, within-session results. They do not measure cross-person or cross-device generalization. Disjoint image identities do not erase physiological overlap between successive presentations.
unique = np.unique(labels)
np.random.default_rng(2026).shuffle(unique)
first, second = int(0.6 * len(unique)), int(0.8 * len(unique))
train_ids, valid_ids, test_ids = [
np.sort(group) for group in np.split(unique, [first, second])
]
train, valid, test = [
np.isin(labels, group) for group in [train_ids, valid_ids, test_ids]
]
Tune the decoder and rank held-out candidate images#
Ridge predicts the frozen image coordinates from EEG. Its pipeline estimates feature means and scales on training images’ responses only. Validation retrieval selects among the three prespecified penalties; the selected fitted model is then evaluated once on test responses without refitting on validation.
Validation responses rank against the 12 validation images; final test
responses rank against all 12 test images. Training images are absent from
both candidate pools. searchsorted maps the actual image IDs to positions
in each sorted gallery; those positions match similarity-matrix columns.
Top-5 counts whether the viewed image occurs among the five highest cosine
similarities. Its uniform reference here is 5/12, not the official gallery’s
reference. The complete test pool is used for every test trial, irrespective
of batching. One participant means this is one participant-level proportion;
a multi-participant evaluation would average participant scores equally.
best_score, best_model = -np.inf, None
for alpha in [1, 10, 100]:
model = make_pipeline(StandardScaler(), Ridge(alpha=alpha))
model.fit(X[train], embeddings[labels[train]])
score = top_k_accuracy_score(
np.searchsorted(valid_ids, labels[valid]),
cosine_similarity(model.predict(X[valid]), embeddings[valid_ids]),
k=5,
labels=np.arange(len(valid_ids)),
)
if score > best_score:
best_score, best_model = score, model
score = top_k_accuracy_score(
np.searchsorted(test_ids, labels[test]),
cosine_similarity(best_model.predict(X[test]), embeddings[test_ids]),
k=5,
labels=np.arange(len(test_ids)),
)
print("Encoder:", image_encoder.model_name)
print("Held-out image top-5 accuracy:", score, "test candidates:", len(test_ids))
fig, ax = plt.subplots(figsize=(5, 4))
ax.bar(["uniform retrieval", "ridge"], [5 / len(test_ids), score])
ax.set(ylabel="Test image top-5 accuracy", ylim=(0, 1))
plt.show()
Encoder: facebook/dinov2-small
Held-out image top-5 accuracy: 0.4666666666666667 test candidates: 12
Make the next comparison informative#
Inspect errors per held-out image before concluding that a representation captures visual content. Some images have similar embeddings, and different images may have different numbers of usable responses. Averaging predictions within image or reporting image-wise scores answers a different evaluation question and should be prespecified.
Opt into EEGDASH_DINO_MODEL=giant to compare the official target
encoder on the same image split; budget its checkpoint and CPU cost.
Larger target vectors alone do not establish competition performance. The
next protocol change is to use the release’s original train/test image sets,
train the EEG mapping on separate source recordings, and retain the complete
held-out gallery. The official NeuralBench preparation step builds its target
cache; its full-retrieval subject-aggregated metric is the relevant comparison.
Related split-selection example: Braindecode train, test and tune.
Total running time of the script: (0 minutes 35.803 seconds)
