eegdash.corpus module

Plan, download, verify and hold a corpus of datasets: EEGDashCorpus.

A corpus is a list of dataset ids (or a CSV with a dataset_id column). The files are fetched by EEGDashDataset.download_all(), exactly as for one dataset; this module adds the list of expected files per dataset, a check of the cache against it, and a report:

corpus = EEGDashCorpus(["ds004395", "nm000289"], cache_dir="/data/cache")
corpus.plan()                        # catalogue only: files, bytes, drift
corpus.download(n_jobs=16, n_datasets=4, time_budget=6 * 3600)
corpus.verify(level="hash")          # sizes, then git-annex hashes
corpus.status()                      # offline, from the manifests
for raw in corpus.records(): ...     # every recording, offline

Command line: python -m eegdash.corpus plan|download|verify|status <csv or a,b,c> [--cache-dir ...]; exit code 1 when anything is missing.

Per dataset the cache holds <dataset>/.eegdash/manifest.jsonl (a header, then one line per expected file: path, kind, size, key, required, state) and <dataset>/.eegdash/COMPLETE, written when every required file is there. A keyless file the archive does not hold is absent: reported, not required (on002680 lists an .fdt beside every .set that embeds its samples).

class eegdash.corpus.CorpusReport(action: str, datasets: list[DatasetReport] = <factory>, seconds: float = 0.0)

Bases: object

The result of plan, download, verify or status.

action: str

datasets: list[DatasetReport]

property ok: bool

No problem; and, except after a plan, every required file present.

property problems: list[str]

seconds: float = 0.0

to_dict() → dict[str, Any]

class eegdash.corpus.DatasetReport(dataset_id: str, served_as: str | None = None, state: str = 'unplanned', entries: dict[str, dict]=<factory>, problems: list[str] = <factory>, subjects: int | None = None, size_bytes: int | None = None, drift: dict[str, list[str]] | None=None)

Bases: object

What one method found for one dataset.

dataset_id: str

drift: dict[str, list[str]] | None = None

entries: dict[str, dict]

property ok: bool

problems: list[str]

row() → dict[str, Any]

served_as: str | None = None

size_bytes: int | None = None

state: str = 'unplanned'

subjects: int | None = None

class eegdash.corpus.EEGDashCorpus(source: str | Path | Sequence[str], *, cache_dir: str | Path | None = None, eeg_dash_instance: Any = None)

Bases: object

A corpus of datasets in one EEGDash cache.

Parameters:

• source (str | Path | Sequence[str]) – Dataset ids, or the path of a CSV with a dataset_id column (an n_subjects column is checked against the catalogue when present).

• cache_dir (str | Path | None) – As for EEGDashDataset (EEGDASH_CACHE_DIR, else ./.eegdash_cache).

• eeg_dash_instance (EEGDash | None) – The catalogue client, created when first needed (verify, status and records never need one).

download(*, n_jobs: int = 8, n_datasets: int = 2, subjects: Sequence[str] | None = None, time_budget: float | None = None) → CorpusReport

Plan, fetch what is missing with download_all, then check sizes.

Datasets go smallest first, n_datasets at a time, n_jobs workers in all. A dataset COMPLETE for the current plan whose files are all still at their sizes is skipped without network. No dataset starts after time_budget seconds (those left are deferred). Resuming is free: download_all skips every file already at its key’s size.

ids: list[str]

plan(*, subjects: Sequence[str] | None = None) → CorpusReport

Query the catalogue and write every dataset’s manifest; no download.

A catalogue change (files added, dropped or re-keyed) withdraws COMPLETE; the report’s drift lists the paths.

records() → Iterator[Any]

Every recording (EEGDashRaw) of every dataset, offline.

status() → CorpusReport

Per dataset, from the manifests and COMPLETE markers only (no network, no stat).

verify(*, level: str = 'size', n_jobs: int = 8) → CorpusReport

Check every expected file, offline; write or withdraw COMPLETE.

size compares sizes with the git-annex keys; hash also reads every keyed file (n_jobs at a time: one reader hashed 19 MB/s on Lustre). Directory recordings and the BrainVision data file are checked too.

eegdash.corpus.main(argv: Sequence[str] | None = None) → int