Skip to content

User Environments

Software stacks at CSCS are now accessible through the so-called User Environments (uenv). They replace the previous monolithic software stack containing everything from which one could load any module, with all the infinite potential conflicts it involves. User environments contain the minimal software stack required for a certain activity, say, building and running ICON. They are generated by spack, packed into single squashfs file and then mounted by the user. In a way, they can be considered as poor man's containers.

Main Advantages of Uenvs

To a large degree, we can consider that there is a separation of concerns between software stacks and the machine OS which enables the following:

  • a much reduced maintenance of software stacks
  • system-wide upgrades are limited to the low level OS, so should be way easier, when not transparent to the users
  • upgrading to more recent software stacks or keeping old ones around (think compiler versions) is way easier
  • users (for now C2SM) are able to build their own uenvs
  • quicker access to software compared to classical multi-files directories

CSCS documentation

A description of user environments and the uenv tool can be found in the CSCS Knowledge Base .

Uenvs central Registry and C2SM uenvs

The user environments provided by CSCS are registered in a central database. In the long run we should be able to operate only with those but, at least for the initial period, there might also be the need for uenvs provided by C2SM. These are accessed by their absolute path. All supported uenvs are documented in the corresponding vClsuter section of the dedicated page.

Warning

Old software stack images didn't have a mount point in the metadata which is now required for the new versions of the uenv tool and its --uenv slurm plugin counterpart. If you have images in your local repository that are older than roughly September 5th, please pull them again - it will only update the metadata.

The uenv command line tool

Official documentation

It can be used to

  • query images by metadata: tags, names, target architecture, etc ...
  • mount images and work interactively
  • run a single command with that uenv mounted
  • access uenv modules

For now, uenv isn't available on Balfrin and requires (easy) manual installation : 

git clone https://github.com/eth-cscs/uenv.git && ./uenv/install --local

Quickstart

In the following - using name/version:tag as metadata to target a uenv can be replaced by the absolute path to the image if necessary. - when a mount point can be passed to a command it defaults to a path stored in the uenv metadata, most often /user-environment

Create a local images repository

In order to be able to use uenv images, you first need to pullthem into a local repository. First create the repo with 

uenv repo create

List available images

uenv image find

Pull an image in the local repo

uenv image pull name/version:tag

Start a uenv for interactive use

This will spawn a new shell with the uenv mounted at /mount-point 

uenv start name/version:tag [/mount-point]
Exit with a usual exit or uenv stop.

Use modules

Software in a uenv can be accessed by using it as a spack upstream (see spack-c2sm integration) but also by loading modules. For that, use the --view=modules option: 

uenv start --view=modules name/version:tag [/mount-point]
A usual module avail command would then show you the different modules added by the uenv on top of the original environment

launch a subprocess with a uenv mounted

uenv run name/version:tag[:mount-point] -- my_command
will run my_command (can be executing a script) with the uenv mounted. This is particularly useful inside scripts.

The --uenv SLURM plugin

CSCS developped the --uenv slurm plugin in order to mount uenvs at runtime. Either use the name of a uenv in your local repo or the full absolute path to the uenv.

sbatch --uenv name/version:tag[:/mount/point]
sbatch --uenv /image/path[:/mount/point]

as for the uenv CLI, :/mount/point can always be omitted and defaults to what is found in the uenv metadata.

On Balfrin, the installed plugin version doesn't allow access via uenv names but uenv image inspect can be used to query the corresponding absolute path:

uenv image inspect prgenv-gnu --format="{sqfs}"
/bret/scratch/cscs/bcumming/.uenv-images/images/1736b4bb5ad9b3c5cae8878c71782a8bf2f2f739dbce8e039b629de418cb4dab/store.squashfs

Mount points

Uenvs cannot be mounted anywhere. They are generated with a predefined installation path contained in it, where the user is supposed to mount them. By default the uenv tool and slurm plugin will use the path found in the metadata, ususally /user-environment and sometimes /user-tools. The later path exists for side environments that would potentially need to be mounted simultaneously to a main one.

spack-c2sm integration

User environments are supported in spack-c2sm versions newer than v0.21.1.3. 

git clone --depth 1 --recurse-submodules --shallow-submodules -b v0.21.1.3 https://github.com/C2SM/spack-c2sm.git
Then start the user environment and use it as a spack upstream with 
uenv start name/version:tag /mount/point
source spack-c2sm/setup-env.sh /mount/point

TODO

  • Update official spack-c2sm documentation