Using Climada on the Euler Cluster (ETH internal)

Using Climada on the Euler Cluster (ETH internal)#

Access to Euler#

See https://scicomp.ethz.ch/wiki/Getting_started_with_clusters for details on how to register at and get started with Euler.

For all steps below, first enter the Cluster via SSH.

Installation directory and working directory#

Please, get familiar with the various Euler storage options: https://scicomp.ethz.ch/wiki/Storage_systems.
As a general rule: use /cluster/project for installation and /cluster/work for data processing.

For ETH WCR group members, the suggested installation and working directories are /cluster/project/climate/$USER and /cluster/work/climate/$USER respectively.
You may have to create the installation directory:

mkdir -p /cluster/project/climate/$USER \
         /cluster/work/climate/$USER

Climada installation in a virtual environment#

1. Load dependencies#

module load \
  gcc/12.2.0 \
  stack/2024-06 \
  python/3.11.6 \
  hdf5/1.14.3 \
  geos/3.9.1 \
  sqlite/3.43.2 \
  eccodes/2.25.0 \
  gdal/3.6.3 \
  eth_proxy

module load proj
module unload proj

(The last two lines may seem odd but they are working around a conficting dependency version situation.)

You need to execute this every time you login to Euler before Climada can be used. To safe yourself from doing it manually, append these lines to the ~/.bashrc script, which is automatically executed upon logging in to Euler.

2. Create and prepare virtual environment#

envname=climada_env

# create environment
python -m venv --system-site-packages /cluster/project/climate/$USER/venv/$envname

# acitvate it
. /cluster/project/climate/$USER/venv/$envname/bin/activate

# install python kernel (to be used in JupyterHub, s.b.)
python -m ipykernel install --user --name $envname

3. Install dependencies#

pip install \
  dask[dataframe] \
  fiona==1.9 \
  gdal==3.6 \
  netcdf4==1.6.2 \
  rasterio==1.4 \
  pyproj==3.7 \
  geopandas==1.0 \
  xarray==2024.9 \
  sparse==0.15

4. Install climada#

There are two options. Either install from the downloaded repository (option A), or use a particular released version (option B).

option A#

cd /cluster/project/climate/$USER  # or wherever you plan to download the repository
git clone https://github.com/CLIMADA-project/climada_python.git  # unless this has been done before
cd climada_python
pip install -e .

option B#

pip install climada=5.0

or whatever version you prefer

5. Adjust the Climada configuration#

Edit a configuration file according to your needs (see Guide_Configuration). Create a climada.conf file e.g., in /cluster/home/$USER/.config with the following content:

{
    "local_data": {
        "system": "/cluster/work/climate/USERNAME/climada/data",
        "demo": "/cluster/project/climate/USERNAME/climada/data/demo",
        "save_dir": "/cluster/work/climate/USERNAME/climada/results"
    }
}

(Replace USERNAME with your nethz-id.)

6. Test the installation#

Check installation in login node:

python -m unittest climada.engine.test.test_impact_calc

This should prompt the usual “OK” in the end. Once that succeeded you may want to test the installation also in a compute node, just for the sake of it:

sbatch --wrap="python -m unittest climada.engine.test.test_impact_calc"

Look for the “OK” in the hereby created slurm-[XXXXXXX].out file

Please see the docs at https://slurm.schedmd.com/ on how to use the slurm batch system and the Wiki https://scicomp.ethz.ch/wiki/Transition_from_LSF_to_Slurm for a mapping of lsf commands to their slurm equivalents.

Run a Jupyter Notebook on Euler#

It is possible to run a Jupyter Notebook on Euler within a JupytherHub instance running as an interactive slurm job. See the documentation https://scicomp.ethz.ch/wiki/JupyterHub.

For using climada inside the jupyter notebook, you need to create a customized jupyterlabrc file by running the following lines:

mkdir -p ~/.config/euler/jupyterhub
cat > ~/.config/euler/jupyterhub/jupyterlabrc <<EOF

module purge

module load \
  gcc/12.2.0 \
  stack/2024-06 \
  python/3.11.6 \
  hdf5/1.14.3 \
  geos/3.9.1 \
  sqlite/3.43.2 \
  eccodes/2.25.0 \
  gdal/3.6.3 \
  r/4.3.2 \
  eth_proxy

module load proj

module unload proj

EOF

Check for the presence of a ~/.config/euler/jupyterhub/jupyterlab.sh file or link. If it is missing, run

ln -s /cluster/software/others/services/jupyterhub/scripts/jupyterlab.sh ~/.config/euler/jupyterhub/jupyterlab.sh

Now you can start a jupyter notebook running on Euler at https://jupyter.euler.hpc.ethz.ch/

Once the Jupyterlab has started, open a Python notebook and select the kernel $envname, which was created in step 2.

Trouble shooting#

1. Python Module not found or available#

  • Make sure your python environment is activated.

  • Run pip install --upgrade MISSING_MODULE.

2. Upgrading from Python 3.9 or 3.10#

Virtual environments created are i.g. only working for the Python version they were created with. In particular Python kernels from 3.9 environments will fail to connect in a Jupyter notebook on https://jupyter.euler.hpc.ethz.ch/.

  • It’s suggested to create new environments and remove the old kernels from ~/.local/share/jupyter/kernels/.

3. Incompatible GEOS version#

If you get a warning UserWarning: The Shapely GEOS version (3.9.1-CAPI-1.14.2) is incompatible with the GEOS version PyGEOS was compiled with (3.9.1-CAPI-1.14.2). Conversions between both will be slow. or similar (version numbers may vary), updating geopandas can help:

  • Create and activate a virtual environment with venv (s.a.)

  • Run pip install --upgrade geopandas

4. Installation doesn’t work#

If you have additional requirements it may be that the installation process described above is failing. In this case you can run climada from a customized singularity container.

Fall back: Singularity Container#

In case the installation in a virtual environment does not work, e.g,, because some module on Euler in incompatible with additional requirements for Python packages, the last resort is an installation of CLIMADA into a Singularity container.
In general, this is more difficult and time-consuming and easier to get wrong. It also requires a lot of diskspace and produces a high number of files, but it provides more flexibility, as one can install basically anything you want.

To install CLIMADA into a Singularity container, follow these steps:

1. create a recipe#

Create a file recipe.txt with the following content:

Bootstrap: docker
From: nvidia/cuda:12.0.0-devel-ubuntu22.04


%labels
    version="1.0.0"
    description="climada"


%post

    # Install requirements
    apt-get -y update
    DEBIAN_FRONTEND="noninteractive" TZ="Europe/Rome" apt-get -y install tzdata
    apt-get install -y `apt-cache depends openssh-client | awk '/Depends:/{print$2}'`
    apt-get download openssh-client
    dpkg --unpack openssh-client*.deb
    rm /var/lib/dpkg/info/openssh-client.postinst -f
    dpkg --configure openssh-client
    apt-get -y install tk tcl rsync wget curl git patch

    mkdir -p /opt/software

    # Install conda and mamba
    cd /opt/software
    curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
    sh ./Miniconda3-latest-Linux-x86_64.sh -p /opt/software/conda -b
    /opt/software/conda/bin/conda install -y -c conda-forge mamba

    # Create and activate environment
    /opt/software/conda/bin/mamba create -n climada_env python=3.11 --yes
    . /opt/software/conda/etc/profile.d/conda.sh && conda activate climada_env

    # Install jupyter
    python -m pip install jupyterhub jupyterlab


    # Install climada from source
    mkdir -p /opt/climada_workspace
    cd /opt/climada_workspace

    git clone https://github.com/CLIMADA-project/climada_python.git
    cd climada_python
    git checkout develop

    mamba env update -n climada_env -f requirements/env_climada.yml
    python -m pip install -e ./

    # Install climada-petals from source
    cd /opt/climada_workspace

    git clone https://github.com/CLIMADA-project/climada_petals.git
    cd climada_petals
    git checkout develop

    mamba env update -n climada_env -f requirements/env_climada.yml
    python -m pip install -e ./


%environment
    #export LC_ALL=C


%runscript
   . /opt/software/conda/bin/activate && conda activate climada_env
   $@

2. build the container#

  • as it is cpu and memory consuming, run a job

  • the container will be created in the directory climada.sif, it is going to be huge, so it’s best located within the project file system

sbatch \
  --ntasks=1\
  --cpus-per-task=1 \
  --time=1:00:00 \
  --job-name="build-climada-container" \
  --mem-per-cpu=4096 \
  --wrap="singularity build --sandbox /cluster/project/[path/to]/climada.sif recipe.txt"

3. Configure jupyterhub#

create a file ~/.config/euler/jupyterhub/jupyterlabrc with the following content:

#!/bin/bash

# Import required modules
module purge
module load stack/2024-05 gcc/13.2.0 python/3.11.6_cuda eth_proxy


# Setup the required env
export JUPYTER_CONFIG_PATH=$HOME/.jupyterlab:$PYTHON_EULER_ROOT/share/jupyter
export JUPYTER_CONFIG_DIR=$HOME/.jupyterlab
export JUPYTER_PATH=$PYTHON_EULER_ROOT/share/jupyter
export JUPYTERLAB_DIR=$PYTHON_EULER_ROOT/share/jupyter/lab
export JUPYTERLAB_ROOT=$PYTHON_EULER_ROOT

export JUPYTER_HOME=${JUPYTER_HOME:-$HOME}
export JUPYTER_DIR=${JUPYTER_DIR:-/}
export JUPYTER_EXTRA_ARGS=${JUPYTER_EXTRA_ARGS:-}

warn_jupyterhub

sleep 1

echo $PYTHON_EULER_ROOT
echo $JUPYTER_EXTRA_ARGS
echo $PROXY_PORT

export PYTHON_ROOT=/opt/software/conda/envs/climada_env
module purge

export APPTAINER_BIND="/cluster,$TMPDIR,$SCRATCH"
singularity exec --nv \
  --env="NVIDIA_VISIBLE_DEVICES=all" \
  --bind /cluster/project/[path/to]/climada_python:/opt/climada_workspace/climada_python \
  --bind /cluster/project/[path/to]/climada_petals:/opt/climada_workspace/climada_petals \
  /cluster/project/[path/to]/climada.sif \
  /bin/bash <<EOF
. /opt/software/conda/bin/activate && conda activate climada_env
export JUPYTER_CONFIG_PATH=$HOME/.jupyterlab:$PYTHON_ROOT/share/jupyter
export JUPYTER_CONFIG_DIR=$HOME/.jupyterlab
export JUPYTER_PATH=$PYTHON_ROOT/share/jupyter
export JUPYTERLAB_DIR=$PYTHON_ROOT/share/jupyter/lab
export JUPYTERLAB_ROOT=$PYTHON_ROOT
export http_proxy=http://proxy.ethz.ch:3128
export https_proxy=http://proxy.ethz.ch:3128
jupyter lab build
jupyterhub-singleuser \
  --preferred-dir="$JUPYTER_HOME" \
  --notebook-dir="$JUPYTER_DIR" $JUPYTER_EXTRA_ARGS \
  --keyfile="$CONFIG_PATH/jupyter.key" \
  --certfile="$CONFIG_PATH/jupyter.crt" \
  --port="$PROXY_PORT"
EOF

replace the [path/to] bits according to your own directories.

Now you can start a jupyter notebook running on Euler at https://jupyter.euler.hpc.ethz.ch/

Once the Jupyterlab has started, open a Python notebook and stick to the default kernel.

Contents