eegdash.corpus_convert module#
Convert the recordings of a corpus into training samples, one part per shard.
The conversion stage of EEGDashCorpus:
corpus = EEGDashCorpus("eegdash-corpus-v0")
corpus.shards() # the parts, the same in every process
corpus.convert("/data/out", shard=3, n_shards=64) # this process's parts
A shard is one dataset, or some of its subjects; it becomes one part folder
<out>/parts/<dataset>_<kkk>/ holding:
samples.zarr/<g>/data (N, window, C) float32, one row per training sample
samples.zarr/<g>/session (N,) int32, the session each sample came from
samples.zarr/<g>/stats (S, 2, C) float32, mean and std per session
samples.zarr/<g>/positions (C, 3) float32, metres
manifest_part.csv one row per session written
summary.json counts, skipped recordings, dropped channels, the recipe
recipe.json the recipe alone
_DONE written last; a part that has it is never redone
A group <g> is one channel set: every session of the shard whose resolved
channels are the same, names sorted. Signals are microvolts and are not
normalised; the per-session statistics are for the loader.
The recipes are defined in eegdash.corpus_recipes, and nowhere else. The
one named reve is that of the REVE paper (section 3.1.1): EEG channels with a known 3-D position, band-pass 0.5-99.5 Hz,
resampled to 200 Hz, cut into samples of 2000 timesteps, recordings of 10 s or
less dropped. zarr is needed to write a part and is imported only then.
- class eegdash.corpus_convert.Bank(names: Sequence[str], xyz: ndarray)[source]
Bases:
objectElectrode name -> 3-D position (metres), with the naming rules of the corpus.
The rules, applied by
resolve():the bank is case-sensitive, but its case-fold duplicates share coordinates, so a case-insensitive match is safe;
a modality prefix (
EEG Fp1) is dropped when the bare name fails;bare
A1..D32are 10-20 names, but 32 or more of them are the rings of a BioSemi cap:biosemi128_*, orbiosemi160_*/biosemi256_*when the outer rings are there;a zero typed for the letter O (
P03) is tried last;the same electrode twice (
CzandCZ) is kept once.
- classmethod from_braindecode() Bank[source]
REVE’s public position bank, through braindecode’s cached copy.
- classmethod from_json(path: str | Path) Bank[source]
A
{name: [x, y, z]}file in metres, plus the BioSemi templates.
- class eegdash.corpus_convert.ConvertReport(shards: list[ShardReport] = <factory>, seconds: float = 0.0, action: str = 'convert')[source]
Bases:
objectThe result of
EEGDashCorpus.convert().- action: str = 'convert'
- property ok: bool
Every shard converted, now or before, and none without a session.
- seconds: float = 0.0
- shards: list[ShardReport]
- class eegdash.corpus_convert.Recipe(name: str, version: int, sfreq: int, l_freq: float, h_freq: float, h_freq_max_ratio: float, resample_method: str, window: int, min_channels: int, units: str, refused: frozenset[str])[source]
Bases:
objectThe preprocessing a part was converted with; written into every part.
- Parameters:
name (str, int) – What a store is checked against before parts are merged into it.
version (str, int) – What a store is checked against before parts are merged into it.
sfreq (int) – Sampling rate of the samples, Hz. A slower recording is skipped.
l_freq (float) – Band-pass edges, Hz.
h_freq (float) – Band-pass edges, Hz.
h_freq_max_ratio (float) – The upper edge is lowered to this fraction of a recording’s own rate when that is below
h_freq.resample_method (str) – The
methodofmne.io.Raw.resample.window (int) – Timesteps per sample. A recording of window timesteps or fewer after resampling is skipped.
min_channels (int) – A recording with fewer channels of known position is skipped.
units (str) – Unit of the stored signal.
refused (frozenset of str) – Dataset ids never converted: the benchmark’s own sources.
- h_freq: float
- h_freq_max_ratio: float
- l_freq: float
- min_channels: int
- name: str
- refuses(dataset_id: str) bool[source]
Whether dataset_id, or the same dataset under its other archive id, is refused.
- resample_method: str
- sfreq: int
- units: str
- version: int
- window: int
- class eegdash.corpus_convert.Shard(dataset_id: str, index: int, subjects: tuple[str, ...], n_bytes: int = 0)[source]
Bases:
objectOne part to convert: a dataset, or some of its subjects.
- dataset_id: str
- index: int
- n_bytes: int = 0
- property name: str
The part’s folder name,
<dataset>_<kkk>.
- class eegdash.corpus_convert.ShardReport(shard: str, dataset_id: str, state: str = 'problem', summary: dict[str, ~typing.Any]=<factory>, problems: list[str] = <factory>)[source]
Bases:
objectWhat
EEGDashCorpus.convert()did with one shard.- dataset_id: str
- property ok: bool
- shard: str
- state: str = 'problem'
- eegdash.corpus_convert.convert_shard(dataset: str, subjects: Sequence[str] | None, root: Path, out: Path, *, recipe: str | Recipe = 'reve', bank: Bank | None = None, local: bool | None = None, n_jobs: int = 1, compress: bool = True, use_dataset_positions: bool = True, max_records: int | None = None, files: Collection[str] | None = None) dict[source]
Convert one shard into the part folder out; returns its summary.
- Parameters:
dataset (str) – The id written into the part.
subjects (sequence of str | None) – The subject labels of this shard; None for every subject.
root (Path) – The dataset’s BIDS tree (
<cache_dir>/<folder>).out (Path) – The part folder. One that holds
_DONEis left alone and its summary returned; one without is an interrupted run and is removed first.recipe (str | Recipe) – The preprocessing;
"reve"by default.bank (Bank | None) – The position bank; REVE’s public one by default.
local (bool | None) – Read the tree per subject folder with MNE-BIDS (a local dataset) instead of through
EEGDashDataset’s offline mode. By default, whether dataset is a local id (is_local_id()).n_jobs (int) – Workers for MNE’s filter and resample.
compress (bool) – Blosc-zstd chunks; False stores them as they are.
use_dataset_positions (bool) – Place channels the bank does not know from the dataset’s electrodes.tsv; False drops them.
max_records (int | None) – Stop after this many recordings (a dry run).
files (collection of str | None) – The dataset’s recordings, as paths inside it. A recording on disk that is not one of them is left out and listed in the summary (
keep_listed()); None converts every recording under the subject folders.
- eegdash.corpus_convert.get_recipe(recipe: str | Recipe) Recipe[source]
A
Recipe, by name or as given.
- eegdash.corpus_convert.is_local_id(dataset_id: str) bool[source]
Whether dataset_id names a local dataset,
<name>-<version>.Such a dataset is not in the catalogue (a gated one, or the user’s own): its BIDS tree is read from
<cache_dir>/<dataset_id>/and is never queried or downloaded.
- eegdash.corpus_convert.keep_listed(recs: Sequence[Any], files: Collection[str] | None) tuple[list[Any], list[dict]][source]
Split recs into the recordings to convert and those left out, with the reason.
A recording is a dataset’s own only under a subject folder of its root:
sourcedata/holds the files a dataset was made from (the same signal a second time),derivatives/what was computed from it. When files is given (the recordings the catalogue or a shipped file list names), a file on disk that is not in it is left out too: a cache keeps a file after the archive renames it, and both names would be converted.
- eegdash.corpus_convert.plan_shards(dataset_id: str, subject_bytes: dict[str, int], max_bytes: int) list[Shard][source]
Pack the subjects of a dataset, in sorted order, into shards of at most max_bytes.
A subject larger than max_bytes is a shard of its own. The plan depends only on its arguments, so every process computes the same one.
- eegdash.corpus_convert.preprocess_recording(raw: Any, bank: Bank, recipe: Recipe = Recipe(name='reve', version=1, sfreq=200, l_freq=0.5, h_freq=99.5, h_freq_max_ratio=0.45, resample_method='polyphase', window=2000, min_channels=3, units='uV', refused=frozenset({'nm000110', 'nm000139', 'ds004362', 'nm000112', 'nm000114', 'nm000113', 'nm000109'})), *, n_jobs: int = 1, electrodes: Path | None = None) Prepared | Skipped[source]
Apply recipe to one MNE
Raw(not yet loaded).EEG channels only, those with a position (the bank, then electrodes, the recording’s
*_electrodes.tsv: seedataset_positions()), band-pass, resample, microvolts, float32.
- eegdash.corpus_convert.sample_offsets(n_times: int, window: int = 2000) ndarray[source]
Start of every training sample in a session of n_times timesteps.
As REVE’s loader cuts them:
np.arange(start, end - window, window). The exclusive end means a session of exactlyk * windowtimesteps yieldsk - 1samples. Reproduced on purpose, so a store holds precisely the samples REVE trains on.