Jobs

jobs.biofluxes.main(cfg)[source]

Prepare biofluxes files for COSMO simulations.

Copies biofluxes files from the project folder (cfg.vprm['dir']) to the int2lm input folder on scratch (cfg.int2lm_input/vprm).

Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes.

jobs.check_output.main(cfg)[source]

Check output variables for physical reasonability and create plots.

This function checks the output variables to ensure they are in a physically reasonable range. It stores the time series of the minimum, maximum, mean, and standard deviation of the variables as a pandas object into a pickle file.

It also creates per-variable plots from the stored time series data.

Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes.

jobs.cosmo.main(cfg)[source]

Setup the namelists for a COSMO run and submit the job to the queue.

Create necessary directory structure to run COSMO (run, output, and restart directories, defined in cfg.cosmo_run, cfg.cosmo_output, and cfg.cosmo_restart_out).

Copy the COSMO-executable from cfg.cosmo['binary_file'] to cfg.cosmo_run/cfg.cosmo['execname'].

Convert the tracer csv file to a COSMO namelist file.

Format the COSMO namelist templates using the information in cfg.

Format the runscript template and submit the job.

Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes.

jobs.emissions.main(cfg)[source]

Copy emission files to the int2lm input directory.

Necessary for both COSMO and COSMOART simulations.

Copy emission files from project folder (cfg.emissions['dir']) to int2lm input folder on scratch (cfg.int2lm_input/emissions).

For COSMO simulations, converts the netCDF-variable-names from string to char (necessary for int2lm).

If there are multiple emission-datasets (cfg.emissions[‘dir’] is a list of paths), they are copied as follows:

cfg.emissions['dir'][0]/cfg.emissions['gridname'][0]YYYYMMDD.nc -> int2lm_input/emissions/emis_YYYYMMDD.nc
cfg.emissions['dir'][1]/cfg.emissions['gridname'][1]YYYYMMDD.nc -> int2lm_input/emissions2/emis_YYYYMMDD.nc
cfg.emissions['dir'][2]/cfg.emissions['gridname'][2]YYYYMMDD.nc -> int2lm_input/emissions3/emis_YYYYMMDD.nc
Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes.

jobs.icon.main(cfg)[source]

Setup the namelists for an ICON run and submit the job to the queue.

Copy the ICON-executable from cfg.icon_binary_file to cfg.icon_work/icon.exe.

Format the ICON-namelist-templates: icon_master.namelist.cfg, icon_NAMELIST_NWP.cfg, using the information in cfg.

Format the runscript-template and submit the job.

Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes.

jobs.icontools.main(cfg)[source]

  • Add GEOSP to all meteo files

  • Submit the runscript for the DWD ICON tools to remap the meteorological files.

  • All runscripts specified in cfg.icontools_runjobs are submitted.

  • The meteorological files are read from the original input directory (cfg.input_root_meteo), and the remapped meteorological files are saved in the input folder on scratch (cfg.icon_input/icbc).

jobs.int2lm.main(cfg)[source]

Setup the namelist for int2lm and submit the job to the queue.

Necessary for both COSMO and COSMOART simulations.

Decide if the soil model should be TERRA or TERRA multi-layer depending on startdate of the simulation.

Create necessary directory structure to run int2lm (run and output directories, defined in cfg.int2lm and cfg.int2lm['output']).

Copy the int2lm-executable from cfg.int2lm['binary_file'] to cfg.int2lm['work']/int2lm.

Copy the extpar-file cfg.int2lm['extpar_file'] to cfg.int2lm_run/work.

COSMOART: Copy the libgrib_api files to cfg.int2lm['work']/libgrib_api.

COSMO: Convert the tracer-csv-files into an int2lm-namelist file.

Format the int2lm-namelist-template using the information in cfg.

Format the runscript-template and submit the job.

Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes.

jobs.obs_nudging.main(cfg)[source]

Copy and rename the obs_nudging files to the COSMO input directory.

In the folder cfg.obs_nudging_dir, the files are saved in the format {prefix}-YYYYMMDD000000-YYYYMMDD000000. COSMO expects files in the format {prefix}x, where x is [nothing], .2, .3, .4, .... This job handles this filename-change and copies them to the COSMO input folder on scratch (cfg.cosmo_input/obs_nudging).

Missing observation files are ignored.

Also copies the blacklist-file blklsttmp.

Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes.

jobs.octe.main(cfg)[source]

Copy necessary input files for COSMO and perturb BG.

Copies the NetCDF-files found at cfg.octe_maps and cfg.octe_lambdas to the COSMO input-directory.

Perturbs the background tracer field. To do that, it reads the lambda-value from the cfg.octe_lambdas (last value along the nparam-dimension) and scales the BG-field produced by int2lm, creating a new variable for each ensemble.

Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes.

jobs.oem.main(cfg)[source]

Copy emission and profile files to the cosmo or icon input directory.

Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes.

Raises:

RuntimeError – If an error occurs during the process.

jobs.online_vprm.main(cfg)[source]

Copy MODIS surface reflectance data and vegetation class fraction file to the cosmo input directory.

Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes.

jobs.photo_rate.main(cfg)[source]

Copy photolysis-rate file to the COSMOART input directory.

Only necessary for COSMOART simulations.

Copy the photolysis-rate file from the project (cfg.photo_rate_file) to the COSMOART input folder on scratch (cfg.cosmo_input/art_photolysis).

Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes.

jobs.post_cosmo.main(cfg)[source]

Copy the output of a COSMO-run to a user-defined position.

Write a runscript to copy all files (COSMO settings & output, int2lm settings, logfiles) from cfg.cosmo_run, cfg.cosmo_output, cfg.int2lm_run, cfg.log_finished_dir to cfg.output_root/.... If the job reduce_output has been run before post_cosmo, a directory cfg.cosmo_output_reduced is created. In this case, cfg.cosmo_output_reduced is copied instead of cfg.cosmo_output.

Submit the job to the xfer-queue.

Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes.

jobs.post_int2lm.main(cfg)[source]

Combine multiple int2lm tracer-output files into a single one for COSMO.

Only necessary for COSMO simulations.

int2lm puts tracers into different netCDF files. Combine the files specified in cfg.post_int2lm_species into a single netCDF file for COSMO.

If cfg.spinup and cfg.post_int2lm_species_spinup are present, also read in the specified variables and take them as an input for COSMO.

Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes.

jobs.prepare_cosmo.main(cfg)[source]

COSMO Data Preparation

This function prepares input data for COSMO simulations by creating necessary directories, copying meteorological files, and handling specific data processing.

  • Copy meteorological files to int2lm input.

  • Create the necessary directory cfg.int2lm_input/meteo.

  • Copy meteorological files from the project directory (cfg.meteo['dir']/cfg.meteo['prefix']YYYYMMDDHH) to the int2lm input folder on scratch (cfg.int2lm_input/meteo).

  • For nested runs (meteorological files are COSMO output: cfg.meteo['prefix'] == 'lffd'), also copy the *c.nc-file with constant parameters.

Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes.

Raises:

RuntimeError – If any subprocess returns a non-zero exit code during execution.

jobs.prepare_icon.main(cfg)[source]

ICON Data Preparation

This function prepares input data for ICON simulations by creating necessary directories, copying meteorological files, and handling specific data processing.

  • Create working directories and copy input files

Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes.

Raises:

RuntimeError – If any subprocess returns a non-zero exit code during execution.

jobs.reduce_output.main(cfg)[source]

Calculates 2D column data and writes them into a new netCDF file. Only a fixed number of levels from COSMO output are considered. Those files are written into a new directory cosmo_output_reduced.

The number of levels is set by the configuration variable cfg.reduce_output['output_levels'] (default = all levels).

Important: If several GRIBOUT sections are used to split the output data, then this code only works in case of the following: The tracers, for which the column-averaged dry-air (X) and moist-air (Y) mole fractions are calculated, have to be

  1. saved in a separate output file and

  2. the output file appears alphabetically after the meteorological variables.

For example, use a GRIBOUT section suffix _met for standard COSMO output, and _trc for tracers.

Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes

jobs.verify_chain.main(cfg)[source]

Compare outputs of the chain to a reference.

Looks for the reference-file in cfg.verify_chain['reference_dir'].

Looks for the output file in cfg.verify_chain['output_dir'] (if not None), else it goes to the output directory created by the COSMO-job.

In the dict cfg.verify_chain['values_to_check'], the user specifies the names of the files to be compared as keys, and the variables to compare as a list.

To compare the temperatures of the last output of the example case, the following variables should be added to the config.yaml file:

verify_chain['reference_dir'] = os.path.join(input_root, "reference_output")
verify_chain['output_dir'] = None
verify_chain['values_to_check'] = {("reference_lffd2015010200.nc","lffd2015010200.nc"):
      ['T']}
Parameters:

cfg (Config) – Object holding all user-configuration parameters as attributes

Tools

The tools are a collection of functions used by the jobs. Most of those functions are well documented and listed here. For others, one may take a look into jobs/tools directly.

jobs.tools.cams4int2cosmo.main(date, inpath, outpath, param)[source]

Prepare CAMS CO2, CO and NOx boundary conditions for int2lm/int2cosmo for the project SMARTCARB.

The CAMS data consists of

The data sets are retrieved as individual 3-hourly files from the MARS archive at ECMWF with names:

cams_0001_2015010500.nc   # for NO and NO2
sfc_0001_2015010500.nc    # for log of surface pressure
cams_gf39_2015010500.nc   # for CO and CO2
sfc_gf39_2015010500.nc    # for log of surface pressure

The path to the directory of the CAMS data can optionally be supplied, otherwise the script should be invoked in the directory of the CAMS data.

The script generates 8 individual 3-hourly IC/BC files:

cams_nox_yyyymmddhh.nc
cams_co2_yyyymmddhh.nc

Usage:

python cams4int2cosmo.py date [-i inpath -o outpath]

Output:

cams_NOX_YYYYMMDD00.nc to cams_NOX_YYYYMMDD21.nc cams_CO2_YYYYMMDD00.nc to cams_CO2_YYYYMMDD21.nc

Parameters:

  • date (str) – date in format YYYYMMDD

  • inpath (str) – path of original CAMS files (default is current path)

  • outpath (str) – path where output files should be generated (default is current path)

  • param (dict) –

jobs.tools.check_model.check_model(cfg, model='COSMO')[source]

Check that the model specified in cfg matched the prescribed model.

Check that cfg.workflow_name == model. If not, raises a value-error. Ignores capitalization of the strings

Parameters:

  • cfg (Config) – Object holding all user-configuration parameters as attributes.

  • model (str) – Prescribed model

jobs.tools.comp_nc.datasets_equal(dataset1, dataset2, variables, verbose=True)[source]

Compare the contents of dataset1 and dataset2

Compare with numpy.isclose whether the two datasets are equal. No check for equality (of the values or bitwise of the files) is performed, as numerical errors can produce slightly different files for essentially identical computations. Rather, the values are compared to absolute and relative tolerances, check np.isclose documentation for more detail.

If variables is not empty, only the provided variables are compared.

Parameters:

  • dataset1 (netCDF4.Dataset) –

  • dataset2 (netCDF4.Dataset) –

  • variables (list of str) – List of the variables to be compared. If it is empty, all variables are compared.

  • verbose (bool) – If True, results will be printed to stdout.

Returns:

True if the datasets, or if provided the selected variables, are equal, False otherwise.

Return type:

bool

jobs.tools.ctnoaa4int2cosmo.main(date, indir, outdir, param)[source]

Process all CarbonTracker files of a single date to a format compatible with int2lm

Parameters:

  • date (str) – date in format YYYYMMDD.

  • indir (str) – directory of original CarbonTracker files (default is current path).

  • outdir (str) – utput directory of processed files (default is indir/processed).

  • param (dict) – dictionary of the parameters

jobs.tools.mozart2int2lm.main(_, infile, outdir, params)[source]

Extract each timeslice from infile into seperate file in outdir.

Copies dimensions, renaming ‘lev’ -> ‘level’.

Copies, renames and combines variables (see source code for specifics).

Transforms ‘lon’ to [-180, 180) range.

Parameters:

  • _ – Ignored

  • infile (str) – Path to file to extract from

  • outdir (str) – Path to the directory where the output files are written to

  • params (dict) – Only entry: ‘suffix’ Processed files are stored as outdir/{suffix}_YYYYMMDDHH.nc

jobs.tools.reduce_output_start_end.main(indir, outdir, strdate_start, strdate_end, nout_levels, csvfile, convert_gas)[source]

Script to reduce output.

Output: TODO

Parameters (TODO)

opathstr

Output path where the processed data is going to be written

jobs.tools.string2char.main(filename)[source]

Convert the variable names of a netcdf-file from strings to a np.array of chars.

Parameters:

filename (str) – Path to the netcdf-file

jobs.tools.vprmsplit.main(year, ipath, opath)[source]

Script to extract VPRM emissions for a single day and to convert the output into an int2lm compatible format.

Output: The script generates individual 1-hour emission files for the specified year:

gpp_yyyymmddhh.nc
ra_yyyymmddhh.nc
Parameters:

  • year (str) – Year (YYYY) of data to be processed The original VPRM input file (e.g., VPRM_ECMWF_*_2017.nc for 2017 fluxes) needs to be present in the input directory ‘ipath’

  • ipath (str) – Input path where the VPRM input file is located

  • opath (str) – Output path where the processed data is going to be written

jobs.tools.write_cosmo_input_ghg.main(csv_filename, namelist_filename, cfg=None)[source]

Convert a table (.csv file) to namelist file (INPUT_GHG) read by COSMO.

Parameters:

  • csv_filename (str) – Path to the source csv-file

  • namelist_filename (str) – Path to the namelist file that will be created

  • cfg (Config) – Object holding all user-configuration parameters as attributes.

jobs.tools.write_int2lm_input_art.main(trcr_filename, set_filename, nml_filename, hstart=0)[source]

Write the INPUT_ART namelist file for int2lm from .csv files

Parameters:

  • trcr_filename (str) – csv file with tracer definitions

  • set_filename (str) – csv file with tracer datasets

  • nml_filename (str) – output filename (INPUT_ART)

  • hstart (int) – meteorology spin up in hours