eegdash.downloader#

File downloading utilities for EEG data from cloud storage.

This module provides functions for downloading EEG data files and BIDS dependencies from AWS S3 storage, with support for caching and progress tracking. It handles the communication between the EEGDash metadata database and the actual EEG data stored in the cloud.

It talks to S3 through a plain synchronous boto3 client rather than s3fs. s3fs transitively pulls in aiobotocore, which hard-pins botocore to a narrow range and makes any environment that also needs boto3 (moabb, awscli, neuralbench, …) effectively unsolvable. All access here is anonymous, read-only, and limited to a handful of operations (HEAD, GET, LIST), so boto3 covers it directly. See eegdash/EEGDash#397.

Functions

annex_key_info(key)

Return (size, algorithm, hexdigest) parsed from a git-annex key.

download_s3_file(s3_path, local_path, *[, ...])

Download a single file from S3 to a local path.

download_files(files, *[, filesystem, ...])

Download multiple S3 URIs to local destinations.

get_s3path(s3_bucket, filepath)

Construct an S3 URI from a bucket and file path.

get_s3_filesystem(*[, max_concurrency, region])

Get an anonymous S3 accessor for public buckets.

verify_file(path, key_or_expected, *[, ...])

Check path against an annex key (or a byte count); raise OSError.

Classes

S3Client(*[, region, max_concurrency])

Minimal anonymous S3 accessor with EEGDash integrity extensions.

eegdash.downloader.annex_key_info(key: str) → tuple[int | None, str | None, str | None][source]

Return (size, algorithm, hexdigest) parsed from a git-annex key.

Keys read <BACKEND>[E]-s<size>[-m<mtime>][-S<chunk>-C<n>]--<hex>[.<ext>]. A value is None when the key does not carry it (WORM/URL backends have no hash) or the string is not a key. A chunk key names one piece of the content, so neither its size nor its hash describes the object and all three are None.

eegdash.downloader.download_s3_file(s3_path: str, local_path: Path, *, filesystem: S3Client | None = None, version_id: str | None = None, annex_key: str | None = None, verify_hash: bool = False) → Path[source]

Download a single file from S3 to a local path.

Handles the download of a raw EEG data file from an S3 bucket, caching it at the specified local path. Creates parent directories if they do not exist. The git-annex key is authoritative, including for twins and version pins. HEAD and GET lengths describe the stored object, which can itself be short.

Parameters:
  • s3_path (str) – The full S3 URI of the file to download.

  • local_path (pathlib.Path) – The local file path where the downloaded file will be saved.

  • filesystem (S3Client | None) – Optional pre-created accessor to reuse across multiple downloads.

  • version_id (str | None) – Select this exact S3 object version for HEAD and GET. A sibling .s3-version.json records the cache identity; unmarked caches are refreshed even when the byte count matches.

  • annex_key (str | None) – git-annex key of the object, giving its size and hash. Defaults to the key the URI ends in (NEMAR …/objects/<key>), if any.

  • verify_hash (bool) – Also check the key’s hash, of a cached file and of a fresh download. Off by default: it reads the whole file.

Returns:

The local path to the downloaded file.

Return type:

pathlib.Path

eegdash.downloader.download_files(files: Sequence[tuple[str, Path]] | Iterable[tuple[str, Path]], *, filesystem: S3Client | None = None, skip_existing: bool = True, skip_missing: bool = False, annex_keys: dict[Path, str] | None = None) → list[Path][source]

Download multiple S3 URIs to local destinations.

Sizes come from the URI or supplied git-annex key when available, else from HEAD. A known key beats the stored object’s own transport length.

Parameters:
  • files (iterable of (str, Path)) – Pairs of (S3 URI, local destination path).

  • filesystem (S3Client | None) – Optional pre-created accessor to reuse across multiple downloads.

  • skip_existing (bool) – If True, do not download files that already exist locally.

  • skip_missing (bool) – If True, skip files that do not exist on S3 instead of raising.

  • annex_keys (dict of Path to str | None) – git-annex key per destination, for URIs that do not end in one (an OpenNeuro twin’s BIDS-named objects); it beats the HEAD size.

eegdash.downloader.get_s3path(s3_bucket: str, filepath: str) → str[source]

Construct an S3 URI from a bucket and file path.

Parameters:
  • s3_bucket (str) – The S3 bucket name (e.g., “s3://my-bucket”).

  • filepath (str) – The path to the file within the bucket.

Returns:

The full S3 URI (e.g., “s3://my-bucket/path/to/file”).

Return type:

str

eegdash.downloader.get_s3_filesystem(*, max_concurrency: int = 20, region: str = 'us-east-2') → S3Client[source]

Get an anonymous S3 accessor for public buckets.

Parameters:
  • max_concurrency (int) – Size of the underlying connection pool (default 20).

  • region (str) – AWS region for the S3 endpoint (default "us-east-2").

Returns:

An anonymous synchronous S3 accessor for EEGDash.

Return type:

S3Client

class eegdash.downloader.S3Client(*, region: str = 'us-east-2', max_concurrency: int = 20)[source]

Bases: object

Minimal anonymous S3 accessor with EEGDash integrity extensions.

Exposes just the operations EEGDash needs — info(), get_file(), ls(), du() — backed by a synchronous boto3 client configured for unsigned (anonymous) access to public buckets. Paths may be given as s3://bucket/key or bucket/key.

info(path: str, *, version_id: str | None = None) → dict[source]

Return object metadata ({"size": <bytes>}) via a HEAD request.

get_file(rpath: str, lpath: str, *, callback=None, version_id: str | None = None, annex_key: str | None = None, verify_hash: bool = False) → None[source]

Download a single object with one unsigned GET (no HEAD, no LIST).

Streams the body to a .part sibling and atomically renames it into place, so an interrupted transfer never leaves a truncated file at the destination (which skip_existing would later mistake for complete). Each chunk’s byte count is fed to callback (a callback(nbytes) callable) for progress.

Error mapping matches the s3fs contract callers depend on: a missing key raises FileNotFoundError, and a 403 — how S3 reports a missing object when anonymous ListBucket is denied, e.g. NEMAR — raises PermissionError. A connection dropped mid-stream is retried (_STREAM_RETRIES attempts) with a fresh GET, as is a 429/500/503 that outlived boto3’s own retries (with backoff). With annex_key the staged file must match the key’s size (and hash, with verify_hash) before it is published: S3’s own length is that of the stored object, which can itself be short (nm000289, 2026-10-16).

ls(path: str, detail: bool = False)[source]

List the immediate objects under a prefix as bucket/key strings.

Lists one directory level (Delimiter="/"), like s3fs.ls — a recursive walk would let a nested subdirectory’s keys be flattened into the parent by callers that take Path(name) (e.g. CTF .ds dirs with a nested hz.ds).

du(path: str) → int[source]

Sum the size in bytes of every object under a prefix (recursive).

eegdash.downloader.verify_file(path: Path, key_or_expected: str | int, *, verify_hash: bool = False, display_path: Path | None = None) → Path[source]

Check path against an annex key (or a byte count); raise OSError.

The size is always checked when known, the key’s hash only with verify_hash (it reads the whole file). A truncated NEMAR object (nm000289, 2026-10-16) passed every check before this one.