Using Climada on the Euler Cluster (ETH internal)¶
1. Access to Euler¶
See https://scicomp.ethz.ch/wiki/Getting_started_with_clusters for details on how to register at and get started with Euler.
For all installation steps, first enter the Cluster via SSH.
2. Installation Directory¶
Please, get familiar with the various Euler storage options: https://scicomp.ethz.ch/wiki/Storage_systems. As a general rule: use /cluster/project for installation and /cluster/work for data processing.
For ETH WCR group members, the suggested installation directory is /cluster/project/climate/USERNAME. You may have to create the insallation directory:
mkdir -p /cluster/project/climate/USERNAME
Then change your working directory:
cd /cluster/project/climate/USERNAME
work/climate/USERNAME is a high performance file-system, optimized for parallel IO and large files, and may be used in case of high performance requirements.
3. Conda Installation¶
Download or update to the latest version of Miniconda. Installation is done by execution of the following steps:
cd /cluster/project/climate/USERNAME
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
miniconda3/bin/conda init
rm Miniconda3-latest-Linux-x86_64.sh
During the installation process of Miniconda, you are prompted to set the working directory according to your choice. Set it to /cluster/project/climate/USERNAME/miniconda3. Once the installation has finished, log out of Euler and in again. The command prompt should be preceded by (base), indicating that the installation was a success and that you login in into conda’s base environment by default.
4. Climada Library¶
Download Climada from the GitHub repository:
cd /cluster/project/climate/USERNAME
git clone https://github.com/CLIMADA-project/climada_python.git
cd climada_python
git checkout develop # or any other branch you intend to work with
5. Climada Environment¶
Create the conda environment:
cd /cluster/project/climate/USERNAME/climada_python
conda env create -f requirements/env_climada.yml --name climada_env
conda env update -n climada_env -f requirements/env_developer.yml
conda activate climada_env
conda install conda-build
conda develop .
Create a climada.conf file in either of
/cluster/home/USERNAME
/cluster/home/USERNAME/climada
/cluster/home/USERNAME/.config
with the following content:
{
"local_data": {
"system": "/cluster/work/climate/USERNAME/climada/data",
"demo": "/cluster/project/climate/USERNAME/climada_python/data/demo",
"save_dir": "/cluster/work/climate/USERNAME/climada/results"
}
}
6. Climada Scripts¶
Create a bash script for executing python scripts in the climada environment, climadajob.sh:
#!/bin/bash
PYTHON_SCRIPT=$1
. ~/.bashrc
conda activate climada_env
python $PYTHON_SCRIPT
echo $PYTHON_SCRIPT completed
Make it executable:
chmod +x climadajob.sh
Create a python script that executes climada code, e.g., climada_smoke_test.py:
from climada import CONFIG, SYSTEM_DIR
from climada.util.test.test_finance import TestNetpresValue
TestNetpresValue().test_net_pres_val_pass()
print(SYSTEM_DIR)
print(CONFIG.local_data.save_dir.str())
7. Job Submission¶
Please see the Wiki: https://scicomp.ethz.ch/wiki/Using_the_batch_system.
With the scripts from above you can submit the python script as a job like this:
bsub [options] /path/to/climadajob.sh /path/to/climada_smoke_test.py
After the job has finished the lsf output file should look something like this:
Sender: LSF System <lsfadmin@eu-ms-010-32>
Subject: Job 161617875: <./climada_job.sh climada_smoke_test.py> in cluster <euler> Done
Job <./climada_job.sh climada_smoke_test.py> was submitted from host <eu-login-41> by user <USERNAME> in cluster <euler> at Thu Jan 28 14:10:15 2021
Job was executed on host(s) <eu-ms-010-32>, in queue <normal.4h>, as user <USERNAME> in cluster <euler> at Thu Jan 28 14:10:42 2021
</cluster/home/USERNAME> was used as the home directory.
</cluster/work/climate/USERNAME> was used as the working directory.
Started at Thu Jan 28 14:10:42 2021
Terminated at Thu Jan 28 14:10:53 2021
Results reported at Thu Jan 28 14:10:53 2021
Your job looked like:
------------------------------------------------------------
# LSBATCH: User input
./climada_job.sh climada_smoke_test.py
------------------------------------------------------------
Successfully completed.
Resource usage summary:
CPU time : 2.99 sec.
Max Memory : 367 MB
Average Memory : 5.00 MB
Total Requested Memory : 1024.00 MB
Delta Memory : 657.00 MB
Max Swap : -
Max Processes : 5
Max Threads : 6
Run time : 22 sec.
Turnaround time : 38 sec.
The output (if any) follows:
/cluster/project/climate/USERNAME/miniconda3/envs/climada/lib/python3.7/site-packages/pandas_datareader/compat/__init__.py:7: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
from pandas.util.testing import assert_frame_equal
/cluster/work/climate/USERNAME/climada/data
/cluster/work/climate/USERNAME/climada/results
python_script.sh completed
Deinstallation
1. Conda¶
Remove the miniconda3 directory from the installation directory:
rm -rf /cluster/project/climate/USERNAME/miniconda3/
Delete the conda related parts from /cluster/home/USERNAME/.bashrc, i.e., everything between
# >>> conda initialize >>># <<< conda initialize <<<2. Climada¶
Remove the climada sources and config file:
rm -rf /cluster/project/climate/USERNAME/climada_python
rm -f /cluster/home/USERNAME/climada.conf /cluster/home/USERNAME/*/climada.conf