fluxnet_shuttle.shuttle

Core functionality for FLUXNET Shuttle operations

module:

fluxnet_shuttle.shuttle

moduleauthor:

Gilberto Z. Pastorello <gzpastorello@lbl.gov>

moduleauthor:

Valerie Hendrix <vchendrix@lbl.gov>

moduleauthor:

Sy-Toan Ngo <sytoanngo@lbl.gov>

platform:

Unix, Windows

created:

2024-10-31

updated:

2025-12-09

This module provides the core functionality for FLUXNET Shuttle operations, including data discovery, download, and source management across multiple FLUXNET data hubs.

The shuttle module serves as the main interface for interacting with different FLUXNET data sources through a unified API.

Metadata Requirements

Metadata fields:

  • SITE_ID

  • SITE_NAME

  • TEAM_MEMBER_NAME

  • TEAM_MEMBER_EMAIL

  • TEAM_MEMBER_ROLE

  • LOCATION_LAT

  • LOCATION_LONG

  • IGBP

  • NETWORK (network affiliations)

  • FLUXNET_PRODUCT_NAME

  • PRODUCT_ID

  • PRODUCT_CITATION

  • PRODUCT_SOURCE_NETWORK

  • FIRST_YEAR

  • LAST_YEAR

  • DOWNLOAD_LINK

  • ONEFLUX_CODE_VERSION

  • DATA_HUB

Functions

async_to_sync(func)

Decorator: If there is no event loop, run synchronously

download([site_ids, snapshot_file, output_dir])

Download FLUXNET data for specified sites using configuration from a snapshot file.

extract_fluxnet_filename_metadata(filename)

Extract product source network, code version, year range, and run from FLUXNET filename.

listall([data_hubs, output_dir])

List all available FLUXNET data from specified data hubs.

unquote(string[, encoding, errors])

Replace %xx escapes by their single-character equivalent.

urlparse(url[, scheme, allow_fragments])

Parse a URL into 6 components: <scheme>://<netloc>/<path>;<params>?<query>#<fragment>

validate_fluxnet_filename_format(filename)

Validate that a filename follows the standard FLUXNET filename format.

Classes

Any(*args, **kwargs)

Special type indicating an unconstrained type.

FluxnetShuttle([data_hubs, config])

Main orchestrator for FLUXNET operations with error collection.

datetime(year, month, day[, hour[, minute[, ...)

The year, month and day arguments are required.

Exceptions

FLUXNETShuttleError

Base error/exception class for FLUXNET Shuttle
fluxnet_shuttle.shuttle.download(site_ids=None, snapshot_file='', output_dir='.', **kwargs)[source]

Download FLUXNET data for specified sites using configuration from a snapshot file.

Downloads are performed concurrently.

Parameters:

  • site_ids (Optional[List[str]]) – List of site IDs to download data for. If None or empty, downloads all sites from snapshot file.

  • snapshot_file (str) – Path to CSV snapshot file containing site configuration

  • output_dir (str) – Directory to save downloaded files (default: current directory)

  • kwargs (Any) – Additional keyword arguments passed to _download_dataset. - user_info: Dictionary with plugin-specific user info (e.g., {“ameriflux”: {…}})

Returns:

List of downloaded filenames

Return type:

list

Raises:

FLUXNETShuttleError – If snapshot_file is invalid or sites not found

fluxnet_shuttle.shuttle.extract_fluxnet_filename_metadata(filename)[source]

Extract product source network, code version, year range, and run from FLUXNET filename.

FLUXNET filenames follow the archive format (ZIP):

<network_id>_<site_id>_FLUXNET_<year_range>_<version>_<run>.zip

Parameters:

filename (str) – The filename or URL to extract metadata from

Return type:

tuple[str, str, int, int, str]

Returns:

Tuple of (product_source_network, oneflux_code_version, first_year, last_year, run). Returns (“”, “”, 0, 0, “”) if filename is invalid.

Examples

>>> extract_fluxnet_filename_metadata("AMF_US-Ha1_FLUXNET_1991-2020_v1.2_r2.zip")
('AMF', 'v1.2', 1991, 2020, 'r2')
>>> extract_fluxnet_filename_metadata("invalid_filename.zip")
('', '', 0, 0, '')
fluxnet_shuttle.shuttle.listall(data_hubs=None, output_dir='.')[source]

List all available FLUXNET data from specified data hubs.

Parameters:

  • data_hubs (Optional[List[str]]) – List of data hub plugin names to include (e.g., [“ameriflux”, “icos”]). If None or empty, all available data hub plugins are included.

  • output_dir (str) – Directory to save the snapshot file (default: current directory)

Returns:

CSV filename containing data availability information

Return type:

str

fluxnet_shuttle.shuttle.validate_fluxnet_filename_format(filename)[source]

Validate that a filename follows the standard FLUXNET filename format.

Valid format (ZIP archive):

<network_id>_<site_id>_FLUXNET_<year_range>_<version>_<run>.zip

Examples: - AMF_US-Ha1_FLUXNET_1991-2020_v1.2_r2.zip - ICOSETC_BE-Bra_FLUXNET_2020-2024_v1.4_r1.zip

Parameters:

filename (str) – The filename (or URL containing filename) to validate

Return type:

bool

Returns:

True if the filename matches the expected format, False otherwise

Examples

>>> validate_fluxnet_filename_format("AMF_US-Ha1_FLUXNET_1991-2020_v1.2_r2.zip")
True
>>> validate_fluxnet_filename_format("invalid_filename.zip")
False
>>> validate_fluxnet_filename_format("AMF_US-Ha1 FLUXNET_1991-2020_v1.2_r2.zip")
False