{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Using Climada on the Euler Cluster (ETH internal)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Content\n", "\n", "1. [Access to Euler](#ssh)\n", "2. [Installation and working directories](#dir)\n", "3. [Pre-installed version of Climada](#pic)\n", " 1. [Load dependencies](#lmod)\n", " 2. [Check installation](#check)\n", " 3. [Adjust the Climada configuration](#conf)\n", " 4. [Run a job](#bsub)\n", "4. [Working with Git Branches](#branch)\n", " 1. [Load dependencies](#lmod2)\n", " 2. [Create installation environment](#venv)\n", " 3. [Check out sources](#clone)\n", " 4. [Pip install Climada](#pip)\n", " 5. [Check installation](#check2)\n", " 6. [Adjust the Climada configuration](#conf2)\n", " 7. [Run a job](#bsub2)\n", "5. [Fallback: Conda installation](#fallb)\n", " 1. [Conda installation](#conda)\n", " 4. [Check out sources](#git)\n", " 3. [Climada environemnt](#env)\n", " 5. [Adjust the Climada configuration](#conf3)\n", " 6. [Climada sripts](#bash)\n", " 7. [Run a job](#bsub3)\n", "6. [Conda Deinstallation](#Rm)\n", " 1. [Conda](#RmConda)\n", " 2. [Climada](#RmClimada)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Access to Euler\n", "\n", "See https://scicomp.ethz.ch/wiki/Getting_started_with_clusters for details on how to register at and get started with Euler.\n", "\n", "For all steps below, first enter the Cluster via SSH." ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "## Installation- and working directories \n", "Please, get familiar with the various Euler storage options: https://scicomp.ethz.ch/wiki/Storage_systems.
\n", "As a general rule: use `/cluster/project` for installation and `/cluster/work` for data processing.\n", "\n", "For ETH WCR group members, the suggested installation and working directories are `/cluster/project/climate/$USER` and `/cluster/work/climate/$USER` respectively.
\n", "You may have to create the insallation directory:\n", "\n", "```bash\n", "mkdir -p /cluster/project/climate/$USER \\\n", " /cluster/work/climate/$USER\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Pre-installed version of Climada\n", "\n", "Climada is pre-installed and available in the default pip environment of Euler." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1. Load dependencies\n", "```bash\n", "env2lmod\n", "module load gcc/6.3.0 python/3.8.5 gdal/3.1.2 geos/3.8.1 proj/7.2.1 libspatialindex/1.8.5 hdf5/1.10.1 netcdf/4.4.1.1 eccodes/2.21.0 zlib/1.2.9\n", "```\n", "\n", "You need to execute these two lines every time you login to Euler before Climada can be used. To safe yourself from doing it manually, one can append these lines to the ~/.bashrc script, which is automatically executed upon logging in to Euler." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. Check installation\n", "```bash\n", "python -c 'import climada; print(climada.__file__)'\n", "```\n", " should output something like this:\n", "```bash\n", "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/climada/__init__.py\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3. Adjust the Climada configuration\n", "Edit a configuration file according to your needs (see [Guide_Configuration](./Guide_Configruation.ipynb)).\n", "Create a climada.conf file e.g., in /cluster/home/$USER/.config with the following content:\n", "\n", "```json\n", "{\n", " \"local_data\": {\n", " \"system\": \"/cluster/work/climate/USERNAME/climada/data\",\n", " \"demo\": \"/cluster/project/climate/USERNAME/climada_python/data/demo\",\n", " \"save_dir\": \"/cluster/work/climate/USERNAME/climada/results\"\n", " }\n", "}\n", "```\n", "(Replace USERNAME with your nethz-id.)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4. Run a job\n", "Please see the Wiki: https://scicomp.ethz.ch/wiki/Using_the_batch_system for an overview on how to use `bsub`.\n", "\n", "```bash\n", "cd /cluster/work/climate/$USER # change to the working directory\n", "bsub [bsub-options*] python climada_job_script.py # submit the job\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Working with Git branches\n", "If the Climada version of the default installation is not according to your needs, you can install Climada from a local Git repository." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1. Load dependencies\n", "See [Load dependencies](#lmod) above." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. Create installation environment\n", "```bash\n", "python -m venv --system-site-packages /cluster/project/climate/$USER/climada_venv\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3. Checkout sources\n", "```bash\n", "cd /cluster/project/climate/$USER\n", "git clone https://github.com/CLIMADA-project/climada_python.git\n", "cd climada_python\n", "git checkout develop # i.e., your branch of interest\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4. Pip install Climada\n", "```bash\n", "source /cluster/project/climate/$USER/climada_venv/bin/activate\n", "pip install -e /cluster/project/climate/$USER/climada_python\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5. Check installation\n", "```bash\n", "cd /cluster/work/climate/$USER\n", "python -c 'import climada; print(climada.__file__)'\n", "```\n", "\n", "should output exactly this (with explicit \\$USER):\n", "```bash\n", "/cluster/project/climate/$USER/climada_python/climada/__init__.py\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 6. Adjust the Climada configuration\n", "See [Adjust the Climada configuration](#conf) above." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 7. Run a job\n", "See [Run a job](#bsub) above." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fallback: Conda installation\n", "If Climada cannot be installed through pip because of changed dependency requirements, there is still the possibility to install Climada through the Conda environment.\n", "> **WARNING:** This apporach is highly discouraged, as it imposes a heavy and mostly unnecessary burdon on the file system of the cluster." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1. Conda Installation \n", "\n", "Download or update to the latest version of [Miniconda](https://conda.io/miniconda.html).
\n", "Installation is done by execution of the following steps:\n", "\n", "```bash\n", "cd /cluster/project/climate/USERNAME\n", "wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\n", "bash Miniconda3-latest-Linux-x86_64.sh\n", "miniconda3/bin/conda init\n", "rm Miniconda3-latest-Linux-x86_64.sh\n", "```\n", "\n", "During the installation process of Miniconda, you are prompted to set the working directory according to your choice.\n", "Set it to `/cluster/project/climate/USERNAME/miniconda3`.
\n", "Once the installation has finished, log out of Euler and in again. The command prompt should be preceded by `(base)`, \n", "indicating that the installation was a success and that you login in into conda's base environment by default." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. Checkout sources \n", "See [Checkout sources](#git) above." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3. Climada Environment \n", "\n", "Create the conda environment:\n", "\n", "```bash\n", "cd /cluster/project/climate/USERNAME/climada_python\n", "conda env create -f requirements/env_climada.yml --name climada_env\n", "conda env update -n climada_env -f requirements/env_developer.yml\n", "\n", "conda activate climada_env\n", "conda install conda-build\n", "conda develop .\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4. Adjust the Climada configuration \n", "\n", "See [Adjust the Climada configuration](#conf) above." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5. Climada Scripts \n", "\n", "Create a bash script for executing python scripts in the climada environment, `climadajob.sh`:\n", "```bash\n", "#!/bin/bash\n", "PYTHON_SCRIPT=$1\n", "shift\n", ". ~/.bashrc\n", "conda activate climada_env\n", "python $PYTHON_SCRIPT $@\n", "echo $PYTHON_SCRIPT completed\n", "```\n", "\n", "Make it executable:\n", "\n", "```bash\n", "chmod +x climadajob.sh\n", "```\n", "\n", "Create a python script that executes climada code, e.g., `climada_smoke_test.py`:\n", "\n", "```python\n", "import sys\n", "from climada import CONFIG, SYSTEM_DIR\n", "from climada.util.test.test_finance import TestNetpresValue\n", "TestNetpresValue().test_net_pres_val_pass()\n", "print(SYSTEM_DIR)\n", "print(CONFIG.local_data.save_dir.str())\n", "print(\"the script ran with arguments\", sys.argv)\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 6. Run a Job \n", "\n", "Please see the Wiki: https://scicomp.ethz.ch/wiki/Using_the_batch_system.\n", "\n", "With the scripts from above you can submit the python script as a job like this:\n", "\n", "```bash\n", "bsub [options] /path/to/climadajob.sh /path/to/climada_smoke_test.py arg1 arg2\n", "```\n", "\n", "After the job has finished the lsf output file should look something like this:\n", "\n", "```txt\n", "Sender: LSF System \n", "Subject: Job 161617875: <./climada_job.sh climada_smoke_test.py arg1 arg2> in cluster Done\n", "\n", "Job <./climada_job.sh climada_smoke_test.py arg1 arg2> was submitted from host by user in cluster at Thu Jan 28 14:10:15 2021\n", "Job was executed on host(s) , in queue , as user in cluster at Thu Jan 28 14:10:42 2021\n", " was used as the home directory.\n", " was used as the working directory.\n", "Started at Thu Jan 28 14:10:42 2021\n", "Terminated at Thu Jan 28 14:10:53 2021\n", "Results reported at Thu Jan 28 14:10:53 2021\n", "\n", "Your job looked like:\n", "\n", "------------------------------------------------------------\n", "# LSBATCH: User input\n", "./climada_job.sh climada_smoke_test.py arg1 arg2\n", "------------------------------------------------------------\n", "\n", "Successfully completed.\n", "\n", "Resource usage summary:\n", "\n", " CPU time : 2.99 sec.\n", " Max Memory : 367 MB\n", " Average Memory : 5.00 MB\n", " Total Requested Memory : 1024.00 MB\n", " Delta Memory : 657.00 MB\n", " Max Swap : -\n", " Max Processes : 5\n", " Max Threads : 6\n", " Run time : 22 sec.\n", " Turnaround time : 38 sec.\n", "\n", "The output (if any) follows:\n", "\n", "/cluster/project/climate/USERNAME/miniconda3/envs/climada/lib/python3.7/site-packages/pandas_datareader/compat/__init__.py:7: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n", " from pandas.util.testing import assert_frame_equal\n", "/cluster/work/climate/USERNAME/climada/data\n", "/cluster/work/climate/USERNAME/climada/results\n", "the script ran with arguments ['/path/to/climada_smoke_test.py', 'arg1' 'arg2']\n", "python_script.sh completed\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conda Deinstallation " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1. Conda \n", "\n", "Remove the miniconda3 directory from the installation directory:\n", "\n", "```bash\n", "rm -rf /cluster/project/climate/USERNAME/miniconda3/\n", "```\n", "\n", "Delete the conda related parts from `/cluster/home/USERNAME/.bashrc`, i.e., everything between\n", "\n", "`# >>> conda initialize >>>`\\\n", "and\\\n", "`# <<< conda initialize <<<`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. Climada \n", "\n", "Remove the climada sources and config file:\n", "\n", "```bash\n", "rm -rf /cluster/project/climate/USERNAME/climada_python\n", "rm -f /cluster/home/USERNAME/climada.conf /cluster/home/USERNAME/*/climada.conf\n", "```" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.8" } }, "nbformat": 4, "nbformat_minor": 4 }