eegdash.downloader#

File downloading utilities for EEG data from cloud storage.

This module provides functions for downloading EEG data files and BIDS dependencies from AWS S3 storage, with support for caching and progress tracking. It handles the communication between the EEGDash metadata database and the actual EEG data stored in the cloud.

It talks to S3 through a plain synchronous boto3 client rather than s3fs. s3fs transitively pulls in aiobotocore, which hard-pins botocore to a narrow range and makes any environment that also needs boto3 (moabb, awscli, neuralbench, …) effectively unsolvable. All access here is anonymous, read-only, and limited to a handful of operations (HEAD, GET, LIST), so boto3 covers it directly. See eegdash/EEGDash#397.

Functions

download_s3_file(s3_path, local_path, *[, ...])

Download a single file from S3 to a local path.

download_files(files, *[, filesystem, ...])

Download multiple S3 URIs to local destinations.

get_s3path(s3_bucket, filepath)

Construct an S3 URI from a bucket and file path.

get_s3_filesystem(*[, max_concurrency, region])

Get an anonymous S3 accessor for public buckets.

Classes

S3Client(*[, region, max_concurrency])

Minimal, anonymous S3 accessor with an s3fs-compatible surface.
eegdash.downloader.download_s3_file(s3_path: str, local_path: Path, *, filesystem: S3Client | None = None) → Path[source]

Download a single file from S3 to a local path.

Handles the download of a raw EEG data file from an S3 bucket, caching it at the specified local path. Creates parent directories if they do not exist.

Parameters:

  • s3_path (str) – The full S3 URI of the file to download.

  • local_path (pathlib.Path) – The local file path where the downloaded file will be saved.

  • filesystem (S3Client | None) – Optional pre-created accessor to reuse across multiple downloads.

Returns:

The local path to the downloaded file.

Return type:

pathlib.Path

eegdash.downloader.download_files(files: Sequence[tuple[str, Path]] | Iterable[tuple[str, Path]], *, filesystem: S3Client | None = None, skip_existing: bool = True, skip_missing: bool = False) → list[Path][source]

Download multiple S3 URIs to local destinations.

Parameters:

  • files (iterable of (str, Path)) – Pairs of (S3 URI, local destination path).

  • filesystem (S3Client | None) – Optional pre-created accessor to reuse across multiple downloads.

  • skip_existing (bool) – If True, do not download files that already exist locally.

  • skip_missing (bool) – If True, skip files that do not exist on S3 instead of raising.

eegdash.downloader.get_s3path(s3_bucket: str, filepath: str) → str[source]

Construct an S3 URI from a bucket and file path.

Parameters:

  • s3_bucket (str) – The S3 bucket name (e.g., “s3://my-bucket”).

  • filepath (str) – The path to the file within the bucket.

Returns:

The full S3 URI (e.g., “s3://my-bucket/path/to/file”).

Return type:

str

eegdash.downloader.get_s3_filesystem(*, max_concurrency: int = 20, region: str = 'us-east-2') → S3Client[source]

Get an anonymous S3 accessor for public buckets.

Parameters:

  • max_concurrency (int) – Size of the underlying connection pool (default 20).

  • region (str) – AWS region for the S3 endpoint (default "us-east-2").

Returns:

An anonymous S3 accessor with an s3fs-compatible surface.

Return type:

S3Client

class eegdash.downloader.S3Client(*, region: str = 'us-east-2', max_concurrency: int = 20)[source]

Bases: object

Minimal, anonymous S3 accessor with an s3fs-compatible surface.

Exposes just the operations EEGDash needs — info(), get_file(), ls(), du() — backed by a synchronous boto3 client configured for unsigned (anonymous) access to public buckets. Paths may be given as s3://bucket/key or bucket/key.

info(path: str) → dict[source]

Return object metadata ({"size": <bytes>}) via a HEAD request.

get_file(rpath: str, lpath: str, *, callback=None) → None[source]

Download a single object with one unsigned GET (no HEAD, no LIST).

Streams the body to a .part sibling and atomically renames it into place, so an interrupted transfer never leaves a truncated file at the destination (which skip_existing would later mistake for complete). Each chunk’s byte count is fed to callback (a callback(nbytes) callable) for progress.

Error mapping matches the s3fs contract callers depend on: a missing key raises FileNotFoundError, and a 403 — how S3 reports a missing object when anonymous ListBucket is denied, e.g. NEMAR — raises PermissionError.

ls(path: str, detail: bool = False)[source]

List the immediate objects under a prefix as bucket/key strings.

Lists one directory level (Delimiter="/"), like s3fs.ls — a recursive walk would let a nested subdirectory’s keys be flattened into the parent by callers that take Path(name) (e.g. CTF .ds dirs with a nested hz.ds).

du(path: str) → int[source]

Sum the size in bytes of every object under a prefix (recursive).