eegdash.corpus_merge module#
Merge converted parts into one store, or extend a finished store with them.
The last stage of EEGDashCorpus:
corpus.convert("/data/out") # <out>/parts/<dataset>_<kkk>/
corpus.merge("/data/out", "my-corpus") # <out>/my-corpus.zarr
corpus.merge("/data/out", "v0-plus", base="/data/eegdash-corpus-v0.zarr")
A store <out>/<name>.zarr is a Zarr group whose bucket groups 0 to
n-1 are the groups of the parts (see eegdash.corpus_convert),
numbered in the order of the parts. Its root attributes record the recipe and
index the buckets (buckets). Beside it go <name>-buckets.csv, that
index, and <name>-manifest.csv, one row per session, whose
big_recording_index is the session’s bucket.
With base, a finished store (the published eegdash-corpus-v0, say) is
extended and never written to: its buckets keep their numbers and the parts’
follow. The parts must have been converted with the recipe the base records,
and a dataset the base holds is refused: no dataset is stored twice.
Only the small zarr.json files are copied, so attributes can be set in the
new store without reaching its sources. The chunk files are hard links by
default (link="hard", one file system), else symbolic links or copies.
The store is built as <name>.zarr.partial and renamed once it has passed
its check. zarr is imported only when a merge runs.
- class eegdash.corpus_merge.MergeReport(store: Path, name: str, n_buckets: int = 0, n_samples: int = 0, n_base_buckets: int = 0, datasets: list[str] = <factory>, new_datasets: list[str] = <factory>, link: str = 'hard', seconds: float = 0.0)[source]
Bases:
objectThe result of
merge_parts().- link: str = 'hard'
- n_base_buckets: int = 0
- n_buckets: int = 0
- n_samples: int = 0
- name: str
- seconds: float = 0.0
- store: Path
- eegdash.corpus_merge.bucket_problems(name: str, group: Any, window: int) list[str][source]
What is wrong with one bucket; empty when it agrees with its record of sessions.
Samples, session ids, stats, channels and positions must agree with the
sessionsattribute: a part two workers wrote at once does not (2026-09-19, an older layout).
- eegdash.corpus_merge.merge_parts(parts: Sequence[str | Path], out: str | Path, name: str, *, base: str | Path | None = None, link: str = 'hard', licences: Mapping[str, str] | None = None, overwrite: bool = False) MergeReport[source]
Merge finished parts into the store
<out>/<name>.zarr, after a base’s buckets if given.- Parameters:
parts (sequence of str | Path) – Part folders (each with
_DONE,recipe.jsonandsamples.zarr), in the order their buckets take.out (str | Path) – Where the store and its side files
<name>-buckets.csvand<name>-manifest.csvgo.name (str) – The store’s name.
base (str | Path | None) – A finished store to extend; never written to. Its buckets keep their numbers, its side file
<base name>-manifest.csv(when beside it) heads the new manifest, its licences are carried over.link ({"hard", "symlink", "copy"}) – How the chunk files enter the new store.
licences (mapping of str to str | None) – Dataset id -> licence id, set on the new buckets and added to the root’s
licences.overwrite (bool) – Replace an existing store of that name (once the new one has passed its check).
- Return type:
MergeReport
- Raises:
ValueError – Nothing is written when the merge is refused: an unfinished part, parts of different recipes, a base of another recipe, a benchmark source, a dataset the base holds, a damaged part, an existing store, a store inside the base, an unknown link.