Environment Setup

The following steps guide you through setting up a Python environment for the Processing Chain. All steps only need to be performed once.

pip (Virtual Environment)

Note

Python 3.11 or later is required. cdo and nco are not available via pip and must be provided by the system (e.g. loaded via modules on HPC clusters — see Machine-specific Setup).

Step 1: Load System Python

On HPC systems, load the appropriate modules before continuing. See Machine-specific Setup for machine-specific instructions.

On a generic Linux system with Python 3.11+ already installed, skip directly to Step 2.

Step 2: Create the Virtual Environment

Create the virtual environment at <repo_root>/.venv:

python3 -m venv .venv

Step 3: Activate and Install

Activate the environment and install all Python dependencies:

source .venv/bin/activate
pip install -r requirements.txt

You are now ready to run the chain.

Step 4: Activate in Future Sessions

Each time you open a new session, activate the environment with:

source .venv/bin/activate

You can add this line to your .bashrc to activate it automatically, or use the machine-specific convenience scripts described in Machine-specific Setup.

Step 5: Update the Environment (if needed)

pip install --upgrade -r requirements.txt

Machine-specific Setup

The machines/ directory provides ready-made scripts that load the required system software and optionally activate the Python virtual environment. Each supported machine has its own sub-directory:

machines/
├── euler/
│   ├── modules.sh    # module load commands only
│   └── setup_env.sh  # modules + venv activation (one-stop setup)
└── santis/
    └── setup_env.sh  # uenv start + venv activation

Euler (ETH Zürich)

Euler uses the traditional module system.

1. Load modules (required before using pip or running job scripts):

source machines/euler/modules.sh

This executes:

module load stack/2025-06 gcc/12.2.0 openmpi/4.1.7
module load cdo/2.4.4 nco/5.2.4 netcdf-c/4.9.2
module load python/3.13.0

These versions are kept in sync with the Euler site settings of cesm2icon, so that the provided ICON executable runs in the environment it was built against.

2. One-stop interactive setup (modules + venv activation):

source machines/euler/setup_env.sh

By default, this activates the virtual environment at <repo_root>/.venv. To use a different location, set PROC_CHAIN_VENV before sourcing:

export PROC_CHAIN_VENV=/path/to/your/venv
source machines/euler/setup_env.sh

If the virtual environment does not exist yet, load the modules first and then create it (see pip (Virtual Environment) above):

source machines/euler/modules.sh
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Santis (CSCS)

On Santis, software is provided through uenv (user environments). Because uenv start spawns a new shell, it cannot be sourced inside an existing session.

1. Start an interactive shell with the required environment:

uenv start climtools/25.2:v1 --view=climtools

2. One-stop interactive setup (uenv + venv activation, recommended):

bash machines/santis/setup_env.sh

If you are already inside the uenv and only need to activate the venv:

source machines/santis/setup_env.sh --no-uenv

By default, this activates the virtual environment at <repo_root>/.venv. To use a different location, set PROC_CHAIN_VENV first:

export PROC_CHAIN_VENV=/path/to/your/venv
bash machines/santis/setup_env.sh

Run a single command without entering an interactive shell:

uenv run climtools/25.2:v1 --view=climtools -- ./run_chain.py <casename>

If the virtual environment does not exist yet, enter the uenv first and create it (see pip (Virtual Environment) above):

uenv start climtools/25.2:v1 --view=climtools
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Adding Support for a New Machine

Create a new sub-directory under machines/ following the same pattern:

machines/
└── <machine-name>/
    ├── modules.sh    # load system software (optional)
    └── setup_env.sh  # system software + venv activation

setup_env.sh is the entry point and the only required file. Split the software loading into a separate modules.sh when it is also useful on its own, e.g. inside batch scripts, as on Euler. Use machines/euler/ as a template and adapt the module load commands for the target system.

Store User-specific Data (Optional)

To register your email address and standard compute account, store them in these files in your home directory:

echo <your_account_id> > ~/.acct
echo <your_email_address> > ~/.forward

The Processing Chain reads these files automatically. They can be overridden at any time by setting compute_account and user_mail in your case’s config.yaml.