Skip to content

Backend

The zonda-request repository provides a backend system for processing ICON grid and external parameter requests. It automates the workflow of validating user input, generating grid configurations, running EXTPAR calculations, and archiving results. This backend integrates with GitHub issues and Jenkins pipelines to ensure a seamless and automated process.

Key Components of the Repository

1. Scripts

  • src/hash.py: Generates a unique hash for each request.
  • src/validate_user_input.py: Validates the JSON input provided in the GitHub issue.
  • src/GridExtpar.py: Processes the request by translating the JSON API to Fortran namelists and Python dictionaries (needed for ICON grid generation and EXTPAR)
  • src/archive_artifacts.py: Archives the output files and logs.
  • src/report.py: Posts comments and updates labels on the GitHub issue.

2. Jenkins Pipelines

  • RemoteExtraction: Main pipeline for processing user requests.
  • Cleanup: Cleans up old files and unused resources.
  • Testsuite: Runs tests for multiple issues in parallel.

3. GitHub Integration

  • The repository integrates with GitHub issues to:
    1. Retrieve user requests.
    2. Post updates on the request's status.
    3. Add or remove labels to reflect the current state of the request.

Workflow Summary

  1. A user submits a request via a GitHub issue using the provided template.
  2. The RemoteExtraction pipeline is triggered to process the request.
  3. The pipeline validates the input, generates ICON grids, runs EXTPAR calculations, and archives the results.
  4. The pipeline posts updates to the GitHub issue, including links to the processed data or error logs.
  5. The processed data is available for download for up to 7 days.

Output Files

The provided output folder contains the following files:

  • my_domain_DOMXX.nc: Description of the selected ICON grid (output of ICON Grid Generator).
  • my_domain_DOMXX.parent.nc: Description of the parent grid (RnB(k-1)) of the selected ICON grid (output of ICON Grid Generator with initial_refinement=.TRUE.).
  • my_domain_DOMXX_external_parameter.nc: External parameter data (output of EXTPAR).
  • my_domain_DOMXX_latlon[_rotated].nc: Lat-lon grid corresponding to the selected ICON domain (required by CLM community). Note that the _rotated suffix is added if lrotate=true in the JSON config.
  • my_domain_DOMXX_latlon[_rotated]_reduced.nc: Lat-lon grid corresponding to the selected ICON domain, reduced by 16 cells at each boundary (required by CLM community). Note that the _rotated suffix is added if lrotate=true in the JSON config.
  • my_domain_DOMXX.html: Visualization of the selected ICON grid (my_domain_DOMXX.nc).
  • my_domain_DOMXX_topography.png: Visualization of the topography external parameter stored in my_domain_DOMXX_external_parameter.nc.
  • logs: Collection of EXTPAR log-files (DOMXX_extpar*.log) and the Zonda log-file (gridextpar.log; merging the main ICON Grid Generator and EXTPAR logs).
  • namelists: Collection of namelists for EXTPAR and the ICON Grid Generator.

Remote Server

The remote server is located at ETH Zurich and part of the Institute for Atmospheric and Climate Science (IAC). It has the following specifications:

  • 128 cores
  • 3.0 TB RAM
  • 40 Gbit/s LAN
  • 64-bit Linux system

With the current setting, a maximum of 6 concurrent builds can be run on this machine. In the case of more concurrent requests, a queuing system is used.

Container Tools

The workflow described above makes use of Podman to manage the container images of EXTPAR and ICON Tools. However, for users that do not follow the usual workflow (e.g. DWD), the Zonda backend also supports Apptainer .

To use Apptainer instead of Podman, users can call the script src/GridExtpar.py with an additional flag: --apptainer. Note that before running GridExtpar.py like this, users have to make sure that the Singularity Image Files (SIF) for EXTPAR and ICON Tools exist in the workspace directory passed to GridExtpar.py via the --workspace argument. The SIF files must be named extpar.sif and icon_tools.sif, respectively.