The Zephyr Ecosystem

Zephyr is made up of four repositories that work together to take a user from “I want this subset of this dataset” to a downloaded NetCDF file:

        flowchart LR
    A[iac-metadata] -->|dataset metadata JSON| B[zephyr-frontend]
    B -->|user builds request| C[zephyr-request]
    C -->|processes request| D[zephyr]
    D -->|output .nc files| E[User]

Repository

Purpose

zephyr-frontend

Web application where users browse available datasets and configure their extraction request. Downloads a JSON request file. See Using the Web Interface.

zephyr-request

GitHub Issues-based request tracking. Users submit their JSON file as an issue; a GitHub Actions pipeline processes it on iacdipl-7.ethz.ch and posts a download link back to the issue.

zephyr

Core extraction engine. Reads a JSON request, finds the matching files, extracts data, saves NetCDF output. See Installation and Running an Extraction.

iac-metadata

Scans the datasets on the server and generates the JSON metadata files consumed by the frontend’s dataset browser (available variables, date ranges, file sizes).

Note

🔒 These repositories are private and require access to the C2SM GitHub organization.

How the repos connect

  1. iac-metadata → Frontend (metadata). Scheduled GitHub Actions workflows in iac-metadata perform a daily scan of the datasets on /net/atmos/data and copy the resulting JSON files to /net/co2/c2sm-services/zephyr/file_index, served at https://data.iac.ethz.ch/zephyr/file_index/. The frontend reads these JSONs to populate the dataset browser.

  2. Frontend → Zephyr-request. After configuring a request, users generate a JSON file from the frontend and open an issue in zephyr-request, inserting the JSON.

  3. Zephyr-request → Zephyr. A GitHub Actions pipeline runs launch_zephyr.py on iacdipl-7.ethz.ch, passing the JSON path, the output directory, the zephyr-request version, and the GitHub issue ID. The version and issue ID are embedded as attributes in the output NetCDF files for traceability. A download link is posted back to the issue.

  4. Self-service. Users with access to the IAC servers can skip zephyr-request entirely and run zephyr directly — see Installation and Running an Extraction.

Two ways to get your data

Whichever path you take, you always end up building the same JSON request (see Request JSON Format) — the two options differ only in who runs the extraction:

You want

Do this

No account/login on IAC servers, and are fine posting a GitHub issue

Build a request in zephyr-frontend, download the JSON, open an issue in zephyr-request (Using the Web Interface)

Access to an IAC machine (e.g. co2) and want full control over when it runs

Install zephyr and run launch_zephyr.py yourself (Installation, Running an Extraction)