eegdash.downloader#
File downloading utilities for EEG data from cloud storage.
This module provides functions for downloading EEG data files and BIDS dependencies from AWS S3 storage, with support for caching and progress tracking. It handles the communication between the EEGDash metadata database and the actual EEG data stored in the cloud.
It talks to S3 through a plain synchronous boto3 client rather than s3fs.
s3fs transitively pulls in aiobotocore, which hard-pins botocore to a
narrow range and makes any environment that also needs boto3 (moabb, awscli,
neuralbench, …) effectively unsolvable. All access here is anonymous, read-only,
and limited to a handful of operations (HEAD, GET, LIST), so boto3 covers it
directly. See eegdash/EEGDash#397.
Functions
|
Return |
|
Download a single file from S3 to a local path. |
|
Download multiple S3 URIs to local destinations. |
|
Construct an S3 URI from a bucket and file path. |
|
Get an anonymous S3 accessor for public buckets. |
|
Check path against an annex key (or a byte count); raise |
Classes
|
Minimal anonymous S3 accessor with EEGDash integrity extensions. |
- eegdash.downloader.annex_key_info(key: str) tuple[int | None, str | None, str | None][source]
Return
(size, algorithm, hexdigest)parsed from a git-annex key.Keys read
<BACKEND>[E]-s<size>[-m<mtime>][-S<chunk>-C<n>]--<hex>[.<ext>]. A value isNonewhen the key does not carry it (WORM/URLbackends have no hash) or the string is not a key. A chunk key names one piece of the content, so neither its size nor its hash describes the object and all three areNone.
- eegdash.downloader.download_s3_file(s3_path: str, local_path: Path, *, filesystem: S3Client | None = None, version_id: str | None = None, annex_key: str | None = None, verify_hash: bool = False) Path[source]
Download a single file from S3 to a local path.
Handles the download of a raw EEG data file from an S3 bucket, caching it at the specified local path. Creates parent directories if they do not exist. The git-annex key is authoritative, including for twins and version pins. HEAD and GET lengths describe the stored object, which can itself be short.
- Parameters:
s3_path (str) – The full S3 URI of the file to download.
local_path (pathlib.Path) – The local file path where the downloaded file will be saved.
filesystem (S3Client | None) – Optional pre-created accessor to reuse across multiple downloads.
version_id (str | None) – Select this exact S3 object version for HEAD and GET. A sibling
.s3-version.jsonrecords the cache identity; unmarked caches are refreshed even when the byte count matches.annex_key (str | None) – git-annex key of the object, giving its size and hash. Defaults to the key the URI ends in (NEMAR
…/objects/<key>), if any.verify_hash (bool) – Also check the key’s hash, of a cached file and of a fresh download. Off by default: it reads the whole file.
- Returns:
The local path to the downloaded file.
- Return type:
- eegdash.downloader.download_files(files: Sequence[tuple[str, Path]] | Iterable[tuple[str, Path]], *, filesystem: S3Client | None = None, skip_existing: bool = True, skip_missing: bool = False, annex_keys: dict[Path, str] | None = None) list[Path][source]
Download multiple S3 URIs to local destinations.
Sizes come from the URI or supplied git-annex key when available, else from HEAD. A known key beats the stored object’s own transport length.
- Parameters:
files (iterable of (str, Path)) – Pairs of (S3 URI, local destination path).
filesystem (S3Client | None) – Optional pre-created accessor to reuse across multiple downloads.
skip_existing (bool) – If True, do not download files that already exist locally.
skip_missing (bool) – If True, skip files that do not exist on S3 instead of raising.
annex_keys (dict of Path to str | None) – git-annex key per destination, for URIs that do not end in one (an OpenNeuro twin’s BIDS-named objects); it beats the HEAD size.
- eegdash.downloader.get_s3path(s3_bucket: str, filepath: str) str[source]
Construct an S3 URI from a bucket and file path.
- eegdash.downloader.get_s3_filesystem(*, max_concurrency: int = 20, region: str = 'us-east-2') S3Client[source]
Get an anonymous S3 accessor for public buckets.
- class eegdash.downloader.S3Client(*, region: str = 'us-east-2', max_concurrency: int = 20)[source]
Bases:
objectMinimal anonymous S3 accessor with EEGDash integrity extensions.
Exposes just the operations EEGDash needs —
info(),get_file(),ls(),du()— backed by a synchronousboto3client configured for unsigned (anonymous) access to public buckets. Paths may be given ass3://bucket/keyorbucket/key.- info(path: str, *, version_id: str | None = None) dict[source]
Return object metadata (
{"size": <bytes>}) via a HEAD request.
- get_file(rpath: str, lpath: str, *, callback=None, version_id: str | None = None, annex_key: str | None = None, verify_hash: bool = False) None[source]
Download a single object with one unsigned GET (no HEAD, no LIST).
Streams the body to a
.partsibling and atomically renames it into place, so an interrupted transfer never leaves a truncated file at the destination (whichskip_existingwould later mistake for complete). Each chunk’s byte count is fed tocallback(acallback(nbytes)callable) for progress.Error mapping matches the
s3fscontract callers depend on: a missing key raisesFileNotFoundError, and a 403 — how S3 reports a missing object when anonymousListBucketis denied, e.g. NEMAR — raisesPermissionError. A connection dropped mid-stream is retried (_STREAM_RETRIESattempts) with a fresh GET, as is a 429/500/503 that outlived boto3’s own retries (with backoff). Withannex_keythe staged file must match the key’s size (and hash, withverify_hash) before it is published: S3’s own length is that of the stored object, which can itself be short (nm000289, 2026-10-16).
- ls(path: str, detail: bool = False)[source]
List the immediate objects under a prefix as
bucket/keystrings.Lists one directory level (
Delimiter="/"), likes3fs.ls— a recursive walk would let a nested subdirectory’s keys be flattened into the parent by callers that takePath(name)(e.g. CTF.dsdirs with a nestedhz.ds).
- eegdash.downloader.verify_file(path: Path, key_or_expected: str | int, *, verify_hash: bool = False, display_path: Path | None = None) Path[source]
Check path against an annex key (or a byte count); raise
OSError.The size is always checked when known, the key’s hash only with verify_hash (it reads the whole file). A truncated NEMAR object (nm000289, 2026-10-16) passed every check before this one.