{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Constants and Configuration" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Content\n", "\n", "1. [Constants](#Const)\n", " 1. [Hard Coded](#ConstHard)\n", " 2. [Conigurable](#ConstConf)\n", " 3. [Where to put constants?](#ConstWhere)\n", "2. [Configuration](#Conf)\n", " 1. [Config files](#ConfFiles)\n", " 2. [Accessing configuration values](#ConfAccess)\n", " 3. [Default Configuration](#ConfDefault)\n", " 4. [Test Configuration](#ConfTest)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Constants \n", "Constants are values that, once initialized, are never changed during the runtime of a program. In Python constants are assigned to variables with capital letters by convention, and vice versa, variables with capital letters are supposed to be constants.\n", "\n", "In principle there are about four ways to define a constant's value:\n", "- _hard coding_: the value is defined in the python code directly\n", "- _argument_: the value is taken from an execution argument\n", "- _context_: the value is derived from the environmental context of the exection, e.g., the current working directory or the date-time of execution start.\n", "- _configuation_: read from a file or database\n", "\n", "In CLIMADA, we only use _hard coding_ and _configuration_ to assign values to constants." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.A. Hard Coded \n", "Hard coding constants is the prefered way to deal with strings that are used to identify objects or files." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'well, arh, ...'" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# suboptimal\n", "my_dict = {'x': 4}\n", "if my_dict['x'] > 3:\n", " msg = 'well, arh, ...'\n", "msg" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'yeah!'" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# good\n", "X = 'x'\n", "my_dict = {X: 4}\n", "if my_dict[X] > 3:\n", " msg = 'yeah!'\n", "msg" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "\"this doesn't mean that every string must be a constant\"" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# possibly overdoing it\n", "X = 'x'\n", "Y = \"this doesn't mean that every string must be a constant\"\n", "my_dict = {X: 4}\n", "if my_dict[X] > 3:\n", " msg = Y\n", "msg" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "this does not work\n" ] }, { "data": { "text/plain": [ "0 1\n", "1 2\n", "2 3\n", "Name: x, dtype: int64" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "X = 'x'\n", "df = pd.DataFrame({'x':[1,2,3], 'y':[4,5,6]})\n", "try:\n", " df.X\n", "except:\n", " from sys import stderr; stderr.write(\"this does not work\\n\")\n", "df[X] # this does work but it's less pretty\n", "df.x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.B. Configurable \n", "When it comes to absolute pathes, it is urgently suggested to not use hard coded constant values, for the obvious reasons. But also relative pathes can cause problems. In particular, they may point to a location where the user has not sufficient access permissions. In order to avoid these problems, _all_ pathes constants in CLIMADA are supposed to be defined through configuration.\\\n", " → pathes must be configurable \n", "\n", "The same applies tu urls to external resources, databases or websites. Since they may change at any time, there addresses are supposed to be defined through configuration. Like this it will be possible to access them without the need of tampering with the source code or waiting for a new release.\\\n", " → urls must be configurable \n", "\n", "Another category of constants that should go into the configuration file are system specifications, such as number of CPU's available for CLIMADA or memory settings.\\\n", " → OS settings must be configurable " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.C. Where to put constants? \n", "As a general rule, constants are defined in the module where they intrinsically belong to. If they belong equally to different modules though or they are meant to be used globally, there is the module `climada.util.constants` which is compiling constants CLIMADA wide." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Configuration " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.A. Configuration files \n", "The proper place to define constants that a user may want (or need) to change without changing the CLIMADA installation are the configuration files.\\\n", "These are files in _json_ format with the name `climada.conf`. There is a default config file that comes with the installation of CLIMADA. But it's possible to have several of them. In this case they are complementing one another.\n", "\n", "CLIMADA looks for configuration files upon `import climada`. There are four loacations to look for configuration files:\n", "- `climada/conf`, the installation directory\n", "- `~/climada/conf`, the user's default climada directory\n", "- `~/.config`, the user's configuration directory,\n", "- `.`, the current working directory\n", "\n", "At each location, the path is followed upwards until a file called `climada.conf` is found or the root of the path is reached. Hence, if e.g., `~/climada/climada.conf` is missing but `~/climada.conf` is present, the latter would be read.\n", "\n", "When two config files are defining the same value, the priorities are:\\\n", "`[..]/./climada.conf` > `~/.config/climada.conf` > `~/climada/conf/climada.conf` > `installation_dir/climada/conf/climada.conf`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Format\n", "A configuration file is a JSON file, with the additional restriction, that all keys must be strings without a '.' (dot) character .\\\n", "For configuration valules that belong to a particular module it is suggested to reflect the code repositories file structure in the json object. For example if a configuration for `my_config_value` that blongs to the module `climada.util.dates_times` is wanted, it would be defined as \n", "```json\n", "{\n", " \"util\": {\n", " \"dates_times\": {\n", " \"my_config_value\": 42\n", " }\n", " }\n", "}\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Referenced Configuration Values\n", "Configuration string values can be referenced from other configuration values. E.g.\n", "```json\n", "{\n", " \"a\": \"x\",\n", " \"b\": \"{a}y\"\n", "}\n", "```\n", "In this example \"b\" is eventually resolved to \"xy\"." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.B. Accessing configuration values \n", "Configuration values can be accessed through the (constant) `CONFIG` from the `climada` module:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from climada import CONFIG" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{drought: {resources: {spei_file_url: http://digital.csic.es/bitstream/10261/153475/8}}, landslide: {resources: {opensearch: https://pmmpublisher.pps.eosdis.nasa.gov/opensearch, climatology_monthly: https://svs.gsfc.nasa.gov/vis/a000000/a004600/a004631/frames/9600x5400_16x9_30p/MonthlyClimatology/[01-12]_ClimatologyMonthly_032818_9600x5400.tif}, local_data: .}, relative_cropyield: {local_data: ~/climada/data/ISIMIP_crop}, trop_cyclone: {random_seed: 54}}" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "CONFIG.hazard" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Data Types\n", "The configuration itself and its attributes have the data type `climada.util.config.Config`" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(climada.util.config.Config, climada.util.config.Config)" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "CONFIG.__class__, CONFIG.hazard.trop_cyclone.random_seed.__class__" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The actual configuration values can be accessed as basic types (float, int, str), provided the definition is according to the respective data type:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "54" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "CONFIG.hazard.trop_cyclone.random_seed.int()" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "cannot convert random_seed to str: , not str\n" ] } ], "source": [ "try:\n", " CONFIG.hazard.trop_cyclone.random_seed.str()\n", "except Exception as e:\n", " from sys import stderr; stderr.write(f\"cannot convert random_seed to str: {e}\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "However, configuration string values can be converted to `pathlib.Path` objects if they are pointing to a directory." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "WindowsPath('C:/Users/me/climada/data/ISIMIP_crop')" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "CONFIG.hazard.relative_cropyield.local_data.dir()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that converting a configuration string to a `Path` object like this will create the specified directory on the fly, unless `dir` is called with the parameter `create=False`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.C. Default Configuration \n", "The conifguration file `climada/conf/climada.conf` contains the default configuration.\\\n", "On the top level it has the following attributes\n", "- __local\\_data__: definition of main pathes for accessing and storing CLIMADA related data\n", " - __system__: top directory, where (persistent) climada data is stored\\\n", " default: `~/climada/data`\n", " - __demo__: top directory for data that is downloaded or created in the CLIMADA tutorials\\\n", " default: `~/climada/demo/data`\n", " - __save\\_dir__: directory where transient (non-persistent) data is stored\\\n", " default: `./results`\n", "- __log\\_level__: minimum log level showed by logging, one of DEBUG, INFO, WARNING, ERROR or CRITICAL.\\\n", " default: `INFO`\n", "- __max\\_matrix\\_size__: maximum matrix size that can be used, can be decreased in order to avoid memory issues\\\n", " default: `100000000` (1e8)\n", "- __exposures__: exposures modules specific configuration\n", "- __hazard__: hazard modules specific configuration" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "dict_keys(['_root', '_comment', 'local_data', 'exposures', 'hazard', 'log_level', 'max_matrix_size'])" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "CONFIG.__dict__.keys()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Data Initialization\n", "When `import climada` is executed in a python script or shell, data files from the installation directory are copied to the location specified int the current configuration.\\\n", "This happens only when climada is used for the first time with the current configuration. Subsequent execution will only check for presence of files and won't overwrite existing files." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Thus, the home directory will automatically be populated with a climada directory and several files from the repository when climada is used.\\\n", "To prevent this and keep the home directory clean, create a config file `~/.config/climada.conf` with customized values for `local_data.system` and `local_data.demo`.\\\n", "As an example, a file with the following content would suppress creation of directories and copying of files during execution of CLIMADA code:\n", "```json\n", "{\n", " \"local_data\": {\n", " \"system\": \"/path/to/installation-dir/climada/data/system\",\n", " \"demo\": \"/path/to/installation-dir/climada/data/demo\",\n", " },\n", "}\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.D. Test Configuration \n", "The configuration values for unit and integration tests are not part of the default configuration ([2.C](#ConfDefault)), since they are irrelevant for the regular CLIMADA user and only aimed for developers.\\\n", "The default test configuration is defined in the `climada.conf` file of the installation directory.\n", "This file contains pathes to files that are read during tests. If they are part of the GitHub repository, their path i.g. starts with the `climada` folder within the installation directory:\n", "```bash\n", "{\n", " \"_comment\": \"this is a climada configuration file meant to supersede the default configuration in climada/conf during test\",\n", " \"test_directory\": \"./climada\",\n", " \"test_data\": \"{test_directory}/test/data\",\n", " \"disc_rates\": {\n", " \"test_data\": \"{test_directory}/entity/disc_rates/test/data\"\n", " },\n", " ...\n", "}\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Obviously, the default `test_directory` is given as the relative path to `./climada`. This is fine if (but only if) unit or integration tests are started from the installation directory, which is the case in the automated tests on the CI server.\\\n", "Developers who intend to start a test from another working directory may have to edit this file and replace the relative path with the absolute path to the installation directory:\n", "```json\n", "{\n", " \"_comment\": \"this is a climada configuration file meant to supersede the default configuration in climada/conf during test\",\n", " \"test_directory\": \"/path/to/installation-dir/climada\",\n", " \"test_data\": \"{test_directory}/test/data\",\n", " \"disc_rates\": {\n", " \"test_data\": \"{test_directory}/entity/disc_rates/test/data\"\n", " },\n", " ...\n", "}\n", "```" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.8" } }, "nbformat": 4, "nbformat_minor": 4 }