eegdash.corpus_convert#

Convert the recordings of a corpus into training samples, one part per shard.

The conversion stage of EEGDashCorpus:

corpus = EEGDashCorpus("eegdash-corpus-v0")
corpus.shards()                                  # the parts, the same in every process
corpus.convert("/data/out", shard=3, n_shards=64)  # this process's parts

A shard is one dataset, or some of its subjects; it becomes one part folder <out>/parts/<dataset>_<kkk>/ holding:

samples.zarr/<g>/data       (N, window, C) float32, one row per training sample
samples.zarr/<g>/session    (N,) int32, the session each sample came from
samples.zarr/<g>/stats      (S, 2, C) float32, mean and std per session
samples.zarr/<g>/positions  (C, 3) float32, metres
manifest_part.csv           one row per session written
summary.json                counts, skipped recordings, dropped channels, the recipe
recipe.json                 the recipe alone
_DONE                       written last; a part that has it is never redone

A group <g> is one channel set: every session of the shard whose resolved channels are the same, names sorted. Signals are microvolts and are not normalised; the per-session statistics are for the loader.

The recipes are defined in eegdash.corpus_recipes, and nowhere else. The one named reve is that of the REVE paper (section 3.1.1): EEG channels with a known 3-D position, band-pass 0.5-99.5 Hz, resampled to 200 Hz, cut into samples of 2000 timesteps, recordings of 10 s or less dropped. zarr is needed to write a part and is imported only then.

Functions

convert_shard(dataset, subjects, root, out, *)

Convert one shard into the part folder out; returns its summary.

get_recipe(recipe)

A Recipe, by name or as given.

is_local_id(dataset_id)

Whether dataset_id names a local dataset, <name>-<version>.

keep_listed(recs, files)

Split recs into the recordings to convert and those left out, with the reason.

plan_shards(dataset_id, subject_bytes, max_bytes)

Pack the subjects of a dataset, in sorted order, into shards of at most max_bytes.

preprocess_recording(raw, bank[, recipe, ...])

Apply recipe to one MNE Raw (not yet loaded).

sample_offsets(n_times[, window])

Start of every training sample in a session of n_times timesteps.

Classes

Bank(names, xyz)

Electrode name -> 3-D position (metres), with the naming rules of the corpus.

ConvertReport(shards, seconds, action)

The result of EEGDashCorpus.convert().

Recipe(name, version, sfreq, l_freq, h_freq, ...)

The preprocessing a part was converted with; written into every part.

Shard(dataset_id, index, subjects[, n_bytes])

One part to convert: a dataset, or some of its subjects.

ShardReport(shard, dataset_id, state, ...)

What EEGDashCorpus.convert() did with one shard.

class eegdash.corpus_convert.Bank(names: Sequence[str], xyz: ndarray)[source]

Bases: object

Electrode name -> 3-D position (metres), with the naming rules of the corpus.

The rules, applied by resolve():

  • the bank is case-sensitive, but its case-fold duplicates share coordinates, so a case-insensitive match is safe;

  • a modality prefix (EEG Fp1) is dropped when the bare name fails;

  • bare A1..D32 are 10-20 names, but 32 or more of them are the rings of a BioSemi cap: biosemi128_*, or biosemi160_* / biosemi256_* when the outer rings are there;

  • a zero typed for the letter O (P03) is tried last;

  • the same electrode twice (Cz and CZ) is kept once.

fold: dict[str, int]
classmethod from_braindecode() → Bank[source]

REVE’s public position bank, through braindecode’s cached copy.

classmethod from_json(path: str | Path) → Bank[source]

A {name: [x, y, z]} file in metres, plus the BioSemi templates.

lookup(name: str) → int | None[source]

Row of name, exact spelling first, then ignoring case.

resolve(ch_names: list[str]) → tuple[list[int], list[str], list[str]][source]

Return (indices kept, their bank names in the same order, names dropped).

positions(canonical: list[str]) → ndarray[source]

The (C, 3) positions of bank names, in the order given.

class eegdash.corpus_convert.ConvertReport(shards: list[ShardReport] = <factory>, seconds: float = 0.0, action: str = 'convert')[source]

Bases: object

The result of EEGDashCorpus.convert().

shards: list[ShardReport]
seconds: float = 0.0
action: str = 'convert'
property problems: list[str]
property ok: bool

Every shard converted, now or before, and none without a session.

to_dict() → dict[str, Any][source]
class eegdash.corpus_convert.Recipe(name: str, version: int, sfreq: int, l_freq: float, h_freq: float, h_freq_max_ratio: float, resample_method: str, window: int, min_channels: int, units: str, refused: frozenset[str])[source]

Bases: object

The preprocessing a part was converted with; written into every part.

Parameters:
  • name (str, int) – What a store is checked against before parts are merged into it.

  • version (str, int) – What a store is checked against before parts are merged into it.

  • sfreq (int) – Sampling rate of the samples, Hz. A slower recording is skipped.

  • l_freq (float) – Band-pass edges, Hz.

  • h_freq (float) – Band-pass edges, Hz.

  • h_freq_max_ratio (float) – The upper edge is lowered to this fraction of a recording’s own rate when that is below h_freq.

  • resample_method (str) – The method of mne.io.Raw.resample.

  • window (int) – Timesteps per sample. A recording of window timesteps or fewer after resampling is skipped.

  • min_channels (int) – A recording with fewer channels of known position is skipped.

  • units (str) – Unit of the stored signal.

  • refused (frozenset of str) – Dataset ids never converted: the benchmark’s own sources.

name: str
version: int
sfreq: int
l_freq: float
h_freq: float
h_freq_max_ratio: float
resample_method: str
window: int
min_channels: int
units: str
refused: frozenset[str]
refuses(dataset_id: str) → bool[source]

Whether dataset_id, or the same dataset under its other archive id, is refused.

to_dict() → dict[str, Any][source]

The recipe as JSON-ready values.

class eegdash.corpus_convert.Shard(dataset_id: str, index: int, subjects: tuple[str, ...], n_bytes: int = 0)[source]

Bases: object

One part to convert: a dataset, or some of its subjects.

dataset_id: str
index: int
subjects: tuple[str, ...]
n_bytes: int = 0
property name: str

The part’s folder name, <dataset>_<kkk>.

class eegdash.corpus_convert.ShardReport(shard: str, dataset_id: str, state: str = 'problem', summary: dict[str, ~typing.Any]=<factory>, problems: list[str] = <factory>)[source]

Bases: object

What EEGDashCorpus.convert() did with one shard.

shard: str
dataset_id: str
state: str = 'problem'
summary: dict[str, Any]
problems: list[str]
property ok: bool
row() → dict[str, Any][source]
eegdash.corpus_convert.convert_shard(dataset: str, subjects: Sequence[str] | None, root: Path, out: Path, *, recipe: str | Recipe = 'reve', bank: Bank | None = None, local: bool | None = None, n_jobs: int = 1, compress: bool = True, use_dataset_positions: bool = True, max_records: int | None = None, files: Collection[str] | None = None) → dict[source]

Convert one shard into the part folder out; returns its summary.

Parameters:
  • dataset (str) – The id written into the part.

  • subjects (sequence of str | None) – The subject labels of this shard; None for every subject.

  • root (Path) – The dataset’s BIDS tree (<cache_dir>/<folder>).

  • out (Path) – The part folder. One that holds _DONE is left alone and its summary returned; one without is an interrupted run and is removed first.

  • recipe (str | Recipe) – The preprocessing; "reve" by default.

  • bank (Bank | None) – The position bank; REVE’s public one by default.

  • local (bool | None) – Read the tree per subject folder with MNE-BIDS (a local dataset) instead of through EEGDashDataset’s offline mode. By default, whether dataset is a local id (is_local_id()).

  • n_jobs (int) – Workers for MNE’s filter and resample.

  • compress (bool) – Blosc-zstd chunks; False stores them as they are.

  • use_dataset_positions (bool) – Place channels the bank does not know from the dataset’s electrodes.tsv; False drops them.

  • max_records (int | None) – Stop after this many recordings (a dry run).

  • files (collection of str | None) – The dataset’s recordings, as paths inside it. A recording on disk that is not one of them is left out and listed in the summary (keep_listed()); None converts every recording under the subject folders.

eegdash.corpus_convert.get_recipe(recipe: str | Recipe) → Recipe[source]

A Recipe, by name or as given.

eegdash.corpus_convert.is_local_id(dataset_id: str) → bool[source]

Whether dataset_id names a local dataset, <name>-<version>.

Such a dataset is not in the catalogue (a gated one, or the user’s own): its BIDS tree is read from <cache_dir>/<dataset_id>/ and is never queried or downloaded.

eegdash.corpus_convert.keep_listed(recs: Sequence[Any], files: Collection[str] | None) → tuple[list[Any], list[dict]][source]

Split recs into the recordings to convert and those left out, with the reason.

A recording is a dataset’s own only under a subject folder of its root: sourcedata/ holds the files a dataset was made from (the same signal a second time), derivatives/ what was computed from it. When files is given (the recordings the catalogue or a shipped file list names), a file on disk that is not in it is left out too: a cache keeps a file after the archive renames it, and both names would be converted.

eegdash.corpus_convert.plan_shards(dataset_id: str, subject_bytes: dict[str, int], max_bytes: int) → list[Shard][source]

Pack the subjects of a dataset, in sorted order, into shards of at most max_bytes.

A subject larger than max_bytes is a shard of its own. The plan depends only on its arguments, so every process computes the same one.

eegdash.corpus_convert.preprocess_recording(raw: Any, bank: Bank, recipe: Recipe = Recipe(name='reve', version=1, sfreq=200, l_freq=0.5, h_freq=99.5, h_freq_max_ratio=0.45, resample_method='polyphase', window=2000, min_channels=3, units='uV', refused=frozenset({'nm000110', 'nm000139', 'ds004362', 'nm000112', 'nm000114', 'nm000113', 'nm000109'})), *, n_jobs: int = 1, electrodes: Path | None = None) → Prepared | Skipped[source]

Apply recipe to one MNE Raw (not yet loaded).

EEG channels only, those with a position (the bank, then electrodes, the recording’s *_electrodes.tsv: see dataset_positions()), band-pass, resample, microvolts, float32.

eegdash.corpus_convert.sample_offsets(n_times: int, window: int = 2000) → ndarray[source]

Start of every training sample in a session of n_times timesteps.

As REVE’s loader cuts them: np.arange(start, end - window, window). The exclusive end means a session of exactly k * window timesteps yields k - 1 samples. Reproduced on purpose, so a store holds precisely the samples REVE trains on.