Constants and Configuration

Content

1. Constants

   1. Hard Coded

   2. Conigurable

   3. Where to put constants?

2. Configuration

   1. Config files

   2. Accessing configuration values

   3. Default Configuration

   4. Test Configuration

1. Constants

Constants are values that, once initialized, are never changed during the runtime of a program. In Python constants are assigned to variables with capital letters by convention, and vice versa, variables with capital letters are supposed to be constants.

In principle there are about four ways to define a constant’s value: - hard coding: the value is defined in the python code directly - argument: the value is taken from an execution argument - context: the value is derived from the environmental context of the exection, e.g., the current working directory or the date-time of execution start. - configuation: read from a file or database

In CLIMADA, we only use hard coding and configuration to assign values to constants.

1.A. Hard Coded

Hard coding constants is the prefered way to deal with strings that are used to identify objects or files.

# suboptimal
my_dict = {'x': 4}
if my_dict['x'] > 3:
    msg = 'well, arh, ...'
msg

'well, arh, ...'

# good
X = 'x'
my_dict = {X: 4}
if my_dict[X] > 3:
    msg = 'yeah!'
msg

'yeah!'

# possibly overdoing it
X = 'x'
Y = "this doesn't mean that every string must be a constant"
my_dict = {X: 4}
if my_dict[X] > 3:
    msg = Y
msg

"this doesn't mean that every string must be a constant"

import pandas as pd
X = 'x'
df = pd.DataFrame({'x':[1,2,3], 'y':[4,5,6]})
try:
    df.X
except:
    from sys import stderr; stderr.write("this does not work\n")
df[X] # this does work but it's less pretty
df.x

this does not work

0    1
1    2
2    3
Name: x, dtype: int64

1.B. Configurable

When it comes to absolute pathes, it is urgently suggested to not use hard coded constant values, for the obvious reasons. But also relative pathes can cause problems. In particular, they may point to a location where the user has not sufficient access permissions. In order to avoid these problems, all pathes constants in CLIMADA are supposed to be defined through configuration.

→ pathes must be configurable

The same applies tu urls to external resources, databases or websites. Since they may change at any time, there addresses are supposed to be defined through configuration. Like this it will be possible to access them without the need of tampering with the source code or waiting for a new release.

→ urls must be configurable

Another category of constants that should go into the configuration file are system specifications, such as number of CPU’s available for CLIMADA or memory settings.

→ OS settings must be configurable

1.C. Where to put constants?

As a general rule, constants are defined in the module where they intrinsically belong to. If they belong equally to different modules though or they are meant to be used globally, there is the module climada.util.constants which is compiling constants CLIMADA wide.

2. Configuration

2.A. Configuration files

The proper place to define constants that a user may want (or need) to change without changing the CLIMADA installation are the configuration files.

These are files in json format with the name climada.conf. There is a default config file that comes with the installation of CLIMADA. But it’s possible to have several of them. In this case they are complementing one another.

CLIMADA looks for configuration files upon import climada. There are four loacations to look for configuration files: - climada/conf, the installation directory - ~/climada/conf, the user’s default climada directory - ~/.config, the user’s configuration directory, - ., the current working directory

At each location, the path is followed upwards until a file called climada.conf is found or the root of the path is reached. Hence, if e.g., ~/climada/climada.conf is missing but ~/climada.conf is present, the latter would be read.

When two config files are defining the same value, the priorities are:

[..]/./climada.conf > ~/.config/climada.conf > ~/climada/conf/climada.conf > installation_dir/climada/conf/climada.conf

A configuration file is a JSON file, with the additional restriction, that all keys must be strings without a ‘.’ (dot) character .

For configuration valules that belong to a particular module it is suggested to reflect the code repositories file structure in the json object. For example if a configuration for my_config_value that blongs to the module climada.util.dates_times is wanted, it would be defined as

{
  "util": {
    "dates_times": {
      "my_config_value": 42
    }
  }
}

Configuration string values can be referenced from other configuration values. E.g.

{
  "a": "x",
  "b": "{a}y"
}

In this example “b” is eventually resolved to “xy”.

2.B. Accessing configuration values

Configuration values can be accessed through the (constant) CONFIG from the climada module:

from climada import CONFIG

CONFIG.hazard

{drought: {resources: {spei_file_url: http://digital.csic.es/bitstream/10261/153475/8}}, landslide: {resources: {opensearch: https://pmmpublisher.pps.eosdis.nasa.gov/opensearch, climatology_monthly: https://svs.gsfc.nasa.gov/vis/a000000/a004600/a004631/frames/9600x5400_16x9_30p/MonthlyClimatology/[01-12]_ClimatologyMonthly_032818_9600x5400.tif}, local_data: .}, relative_cropyield: {local_data: ~/climada/data/ISIMIP_crop}, trop_cyclone: {random_seed: 54}}

The configuration itself and its attributes have the data type climada.util.config.Config

CONFIG.__class__, CONFIG.hazard.trop_cyclone.random_seed.__class__

(climada.util.config.Config, climada.util.config.Config)

The actual configuration values can be accessed as basic types (float, int, str), provided the definition is according to the respective data type:

CONFIG.hazard.trop_cyclone.random_seed.int()

54

try:
    CONFIG.hazard.trop_cyclone.random_seed.str()
except Exception as e:
    from sys import stderr; stderr.write(f"cannot convert random_seed to str: {e}\n")

cannot convert random_seed to str: <class 'int'>, not str

However, configuration string values can be converted to pathlib.Path objects if they are pointing to a directory.

CONFIG.hazard.relative_cropyield.local_data.dir()

WindowsPath('C:/Users/me/climada/data/ISIMIP_crop')

Note that converting a configuration string to a Path object like this will create the specified directory on the fly, unless dir is called with the parameter create=False.

2.C. Default Configuration

The conifguration file climada/conf/climada.conf contains the default configuration.

On the top level it has the following attributes - local_data: definition of main pathes for accessing and storing CLIMADA related data - system: top directory, where (persistent) climada data is stored

default: ~/climada/data - demo: top directory for data that is downloaded or created in the CLIMADA tutorials

default: ~/climada/demo/data - save_dir: directory where transient (non-persistent) data is stored

default: ./results - log_level: minimum log level showed by logging, one of DEBUG, INFO, WARNING, ERROR or CRITICAL.

default: INFO - max_matrix_size: maximum matrix size that can be used, can be decreased in order to avoid memory issues

default: 100000000 (1e8) - exposures: exposures modules specific configuration - hazard: hazard modules specific configuration

CONFIG.__dict__.keys()

dict_keys(['_root', '_comment', 'local_data', 'exposures', 'hazard', 'log_level', 'max_matrix_size'])

When import climada is executed in a python script or shell, data files from the installation directory are copied to the location specified int the current configuration.

This happens only when climada is used for the first time with the current configuration. Subsequent execution will only check for presence of files and won’t overwrite existing files.

Thus, the home directory will automatically be populated with a climada directory and several files from the repository when climada is used.

To prevent this and keep the home directory clean, create a config file ~/.config/climada.conf with customized values for local_data.system and local_data.demo.

As an example, a file with the following content would suppress creation of directories and copying of files during execution of CLIMADA code:

{
    "local_data": {
        "system": "/path/to/installation-dir/climada/data/system",
        "demo": "/path/to/installation-dir/climada/data/demo",
    },
}

2.D. Test Configuration

The configuration values for unit and integration tests are not part of the default configuration (2.C), since they are irrelevant for the regular CLIMADA user and only aimed for developers.

The default test configuration is defined in the climada.conf file of the installation directory. This file contains pathes to files that are read during tests. If they are part of the GitHub repository, their path i.g. starts with the climada folder within the installation directory:

{
    "_comment": "this is a climada configuration file meant to supersede the default configuration in climada/conf during test",
    "test_directory": "./climada",
    "test_data": "{test_directory}/test/data",
    "disc_rates": {
        "test_data": "{test_directory}/entity/disc_rates/test/data"
    },
    ...
}

Obviously, the default test_directory is given as the relative path to ./climada. This is fine if (but only if) unit or integration tests are started from the installation directory, which is the case in the automated tests on the CI server.

Developers who intend to start a test from another working directory may have to edit this file and replace the relative path with the absolute path to the installation directory:

{
    "_comment": "this is a climada configuration file meant to supersede the default configuration in climada/conf during test",
    "test_directory": "/path/to/installation-dir/climada",
    "test_data": "{test_directory}/test/data",
    "disc_rates": {
        "test_data": "{test_directory}/entity/disc_rates/test/data"
    },
    ...
}