{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Using Climada on the Euler Cluster (ETH internal)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Content\n", "\n", "1. [Installation](#Inst)\n", " 1. [Access to Euler](#ssh)\n", " 2. [Installation Directory](#dir)\n", " 3. [Conda Installation](#conda)\n", " 4. [Climada Library](#git)\n", " 5. [Climada Environment](#env)\n", " 6. [Climada Scripts](#bash)\n", " 7. [Job Submission](#job)\n", "2. [Deinstallation](#Rm)\n", " 1. [Conda](#RmConda)\n", " 2. [Climada](#RmClimada)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Installation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1. Access to Euler \n", "\n", "See https://scicomp.ethz.ch/wiki/Getting_started_with_clusters for details on how to register at and get started with Euler.\n", "\n", "For all installation steps, first enter the Cluster via SSH." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. Installation Directory \n", "\n", "Please, get familiar with the various Euler storage options: https://scicomp.ethz.ch/wiki/Storage_systems.
\n", "As a general rule: use `/cluster/project` for installation and `/cluster/work` for data processing.\n", "\n", "For ETH WCR group members, the suggested installation directory is `/cluster/project/climate/USERNAME`.
\n", "You may have to create the insallation directory:\n", "\n", "```bash\n", "mkdir -p /cluster/project/climate/USERNAME\n", "```\n", "\n", "Then change your working directory:\n", "\n", "```bash\n", "cd /cluster/project/climate/USERNAME\n", "```\n", "\n", "`work/climate/USERNAME` is a high performance file-system, optimized for parallel IO and large files, and may be used in case of high performance requirements.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3. Conda Installation \n", "\n", "Download or update to the latest version of [Miniconda](https://conda.io/miniconda.html).
\n", "Installation is done by execution of the following steps:\n", "\n", "```bash\n", "cd /cluster/project/climate/USERNAME\n", "wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\n", "bash Miniconda3-latest-Linux-x86_64.sh\n", "miniconda3/bin/conda init\n", "rm Miniconda3-latest-Linux-x86_64.sh\n", "```\n", "\n", "During the installation process of Miniconda, you are prompted to set the working directory according to your choice.\n", "Set it to `/cluster/project/climate/USERNAME/miniconda3`.
\n", "Once the installation has finished, log out of Euler and in again. The command prompt should be preceded by `(base)`, \n", "indicating that the installation was a success and that you login in into conda's base environment by default." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4. Climada Library \n", "\n", "Download Climada from the GitHub repository:\n", "\n", "```bash\n", "cd /cluster/project/climate/USERNAME\n", "git clone https://github.com/CLIMADA-project/climada_python.git\n", "cd climada_python\n", "git checkout develop # or any other branch you intend to work with\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5. Climada Environment \n", "\n", "Create the conda environment:\n", "\n", "```bash\n", "cd /cluster/project/climate/USERNAME/climada_python\n", "conda env create -f requirements/env_climada.yml --name climada_env\n", "conda env update -n climada_env -f requirements/env_developer.yml\n", "\n", "conda activate climada_env\n", "conda install conda-build\n", "conda develop .\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create a `climada.conf` file in either of\n", "\n", "- /cluster/home/USERNAME\n", "- /cluster/home/USERNAME/climada\n", "- /cluster/home/USERNAME/.config\n", "\n", "with the following content:\n", "\n", "```json\n", "{\n", " \"local_data\": {\n", " \"system\": \"/cluster/work/climate/USERNAME/climada/data\",\n", " \"demo\": \"/cluster/project/climate/USERNAME/climada_python/data/demo\",\n", " \"save_dir\": \"/cluster/work/climate/USERNAME/climada/results\"\n", " }\n", "}\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 6. Climada Scripts \n", "\n", "Create a bash script for executing python scripts in the climada environment, `climadajob.sh`:\n", "```bash\n", "#!/bin/bash\n", "PYTHON_SCRIPT=$1\n", ". ~/.bashrc\n", "conda activate climada_env\n", "python $PYTHON_SCRIPT\n", "echo $PYTHON_SCRIPT completed\n", "```\n", "\n", "Make it executable:\n", "\n", "```bash\n", "chmod +x climadajob.sh\n", "```\n", "\n", "Create a python script that executes climada code, e.g., `climada_smoke_test.py`:\n", "\n", "```python\n", "from climada import CONFIG, SYSTEM_DIR\n", "from climada.util.test.test_finance import TestNetpresValue\n", "TestNetpresValue().test_net_pres_val_pass()\n", "print(SYSTEM_DIR)\n", "print(CONFIG.local_data.save_dir.str())\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 7. Job Submission \n", "\n", "Please see the Wiki: https://scicomp.ethz.ch/wiki/Using_the_batch_system.\n", "\n", "With the scripts from above you can submit the python script as a job like this:\n", "\n", "```bash\n", "bsub [options] /path/to/climadajob.sh /path/to/climada_smoke_test.py\n", "```\n", "\n", "After the job has finished the lsf output file should look something like this:\n", "\n", "```txt\n", "Sender: LSF System \n", "Subject: Job 161617875: <./climada_job.sh climada_smoke_test.py> in cluster Done\n", "\n", "Job <./climada_job.sh climada_smoke_test.py> was submitted from host by user in cluster at Thu Jan 28 14:10:15 2021\n", "Job was executed on host(s) , in queue , as user in cluster at Thu Jan 28 14:10:42 2021\n", " was used as the home directory.\n", " was used as the working directory.\n", "Started at Thu Jan 28 14:10:42 2021\n", "Terminated at Thu Jan 28 14:10:53 2021\n", "Results reported at Thu Jan 28 14:10:53 2021\n", "\n", "Your job looked like:\n", "\n", "------------------------------------------------------------\n", "# LSBATCH: User input\n", "./climada_job.sh climada_smoke_test.py\n", "------------------------------------------------------------\n", "\n", "Successfully completed.\n", "\n", "Resource usage summary:\n", "\n", " CPU time : 2.99 sec.\n", " Max Memory : 367 MB\n", " Average Memory : 5.00 MB\n", " Total Requested Memory : 1024.00 MB\n", " Delta Memory : 657.00 MB\n", " Max Swap : -\n", " Max Processes : 5\n", " Max Threads : 6\n", " Run time : 22 sec.\n", " Turnaround time : 38 sec.\n", "\n", "The output (if any) follows:\n", "\n", "/cluster/project/climate/USERNAME/miniconda3/envs/climada/lib/python3.7/site-packages/pandas_datareader/compat/__init__.py:7: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n", " from pandas.util.testing import assert_frame_equal\n", "/cluster/work/climate/USERNAME/climada/data\n", "/cluster/work/climate/USERNAME/climada/results\n", "python_script.sh completed\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Deinstallation " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1. Conda \n", "\n", "Remove the miniconda3 directory from the installation directory:\n", "\n", "```bash\n", "rm -rf /cluster/project/climate/USERNAME/miniconda3/\n", "```\n", "\n", "Delete the conda related parts from `/cluster/home/USERNAME/.bashrc`, i.e., everything between\n", "\n", "`# >>> conda initialize >>>`\\\n", "and\\\n", "`# <<< conda initialize <<<`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. Climada \n", "\n", "Remove the climada sources and config file:\n", "\n", "```bash\n", "rm -rf /cluster/project/climate/USERNAME/climada_python\n", "rm -f /cluster/home/USERNAME/climada.conf /cluster/home/USERNAME/*/climada.conf\n", "```" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.8" } }, "nbformat": 4, "nbformat_minor": 4 }