Constants and Configuration

Constants and Configuration#

Constants#

Constants are values that, once initialized, are never changed during the runtime of a program. In Python constants are assigned to variables with capital letters by convention, and vice versa, variables with capital letters are supposed to be constants.

In principle there are about four ways to define a constant’s value:

  • hard coding: the value is defined in the python code directly

  • argument: the value is taken from an execution argument

  • context: the value is derived from the environmental context of the execution, e.g., the current working directory or the date-time of execution start.

  • configuration: read from a file or database

In CLIMADA, we only use hard coding and configuration to assign values to constants.

Hard Coded#

Hard coding constants is the preferred way to deal with strings that are used to identify objects or files.

# suboptimal
my_dict = {"x": 4}
if my_dict["x"] > 3:
    msg = "well, arh, ..."
msg

'well, arh, ...'

# good
X = "x"
my_dict = {X: 4}
if my_dict[X] > 3:
    msg = "yeah!"
msg

'yeah!'

# possibly overdoing it
X = "x"
Y = "this doesn't mean that every string must be a constant"
my_dict = {X: 4}
if my_dict[X] > 3:
    msg = Y
msg

"this doesn't mean that every string must be a constant"

import pandas as pd

X = "x"
df = pd.DataFrame({"x": [1, 2, 3], "y": [4, 5, 6]})
try:
    df.X
except:
    from sys import stderr

    stderr.write("this does not work\n")
df[X]  # this does work but it's less pretty
df.x

this does not work

0    1
1    2
2    3
Name: x, dtype: int64

Configurable#

When it comes to absolute paths, it is urgently suggested to not use hard coded constant values, for obvious reasons. But also relative paths can cause problems. In particular, they may point to a location where the user has not sufficient access permissions. In order to avoid these problems, all paths constants in CLIMADA are supposed to be defined through configuration.
→ paths must be configurable

The same applies to urls to external resources, databases or websites. Since they may change at any time, their addresses are supposed to be defined through configuration. Like this it will be possible to access them without the need of tampering with the source code or waiting for a new release.
→ urls must be configurable

Another category of constants that should go into the configuration file are system specifications, such as number of CPU’s available for CLIMADA or memory settings.
→ OS settings must be configurable

Where to put constants?#

As a general rule, constants are defined in the module where they intrinsically belong to. If they belong equally to different modules though or they are meant to be used globally, there is the module climada.util.constants which is compiling constants CLIMADA-wide.

Configuration#

Configuration files#

The proper place to define constants that a user may want (or need) to change without changing the CLIMADA installation are the configuration files.
These are files in json format with the name climada.conf. There is a default config file that comes with the installation of CLIMADA. But it’s possible to have several of them. In this case they are complementing one another.

CLIMADA looks for configuration files upon import climada. There are four locations to look for configuration files:

  • climada/conf, the installation directory

  • ~/climada/conf, the user’s default climada directory

  • ~/.config, the user’s configuration directory,

  • ., the current working directory

At each location, the path is followed upwards until a file called climada.conf is found or the root of the path is reached. Hence, if e.g., ~/climada/climada.conf is missing but ~/climada.conf is present, the latter would be read.

When two config files are defining the same value, the priorities are:
[..]/./climada.conf > ~/.config/climada.conf > ~/climada/conf/climada.conf > installation_dir/climada/conf/climada.conf

../_images/FileSystem-1.png

Format#

A configuration file is a JSON file, with the additional restriction, that all keys must be strings without a ‘.’ (dot) character .
The JSON format looks a lot like a Python dict. But note, that all strings must be surrounded by double quotes and trailing commas are not allowed anywhere.

For configuration values that belong to a particular module it is suggested to reflect the code repositories file structure in the json object. For example, if a configuration for my_config_value that belongs to the module climada.util.dates_times is wanted, it would be defined as

{
  "util": {
    "dates_times": {
      "my_config_value": 42
    }
  }
}

Referenced Configuration Values#

Configuration string values can be referenced from other configuration values. E.g.

{
  "a": "x",
  "b": "{a}y"
}

In this example “b” is eventually resolved to “xy”.

Accessing configuration values#

Configuration values can be accessed through the (constant) CONFIG from the climada module:

from climada import CONFIG

CONFIG.hazard

{trop_cyclone: {random_seed: 54}, storm_europe: {forecast_dir: ./results/forecast/hazards}, test_data: .../climada/hazard/test/data}

Data Types#

The configuration itself and its attributes have the data type climada.util.config.Config

CONFIG.__class__, CONFIG.hazard.trop_cyclone.random_seed.__class__

(climada.util.config.Config, climada.util.config.Config)

The actual configuration values can be accessed as basic types (bool, float, int, str), provided that the definition is according to the respective data type:

CONFIG.hazard.trop_cyclone.random_seed.int()

54

try:
    CONFIG.hazard.trop_cyclone.random_seed.str()
except Exception as e:
    from sys import stderr

    stderr.write(f"cannot convert random_seed to str: {e}\n")

cannot convert random_seed to str: <class 'int'>, not str

However, configuration string values can be converted to pathlib.Path objects if they are pointing to a directory.

CONFIG.hazard.storm_europe.forecast_dir.dir()

Note that converting a configuration string to a Path object like this will create the specified directory on the fly, unless dir is called with the parameter create=False.

Default Configuration#

The conifguration file climada/conf/climada.conf contains the default configuration.
On the top level it has the following attributes:

  • local_data: definition of main paths for accessing and storing CLIMADA related data

    • system: top directory, where (persistent) climada data is stored
      default: ~/climada/data

    • demo: top directory for data that is downloaded or created in the CLIMADA tutorials
      default: ~/climada/demo/data

    • save_dir: directory where transient (non-persistent) data is stored
      default: ./results

  • log_level: minimum log level showed by logging, one of DEBUG, INFO, WARNING, ERROR or CRITICAL.
    default: INFO

  • max_matrix_size: maximum matrix size that can be used, can be decreased in order to avoid memory issues
    default: 100000000 (1e8)

  • exposures: exposures modules specific configuration

  • hazard: hazard modules specific configuration

CONFIG.__dict__.keys()

dict_keys(['_root', '_comment', 'local_data', 'engine', 'exposures', 'hazard', 'util', 'log_level', 'max_matrix_size', 'data_api', 'test_directory', 'test_data', 'disc_rates', 'impact_funcs', 'measures'])

Test Configuration#

The configuration values for unit and integration tests are not part of the default configuration, since they are irrelevant for the regular CLIMADA user and only aimed for developers.
The default test configuration is defined in the climada.conf file of the installation directory. This file contains paths to files that are read during tests. If they are part of the GitHub repository, their path i.g. starts with the climada folder within the installation directory:

{
    "_comment": "this is a climada configuration file meant to supersede the default configuration in climada/conf during test",
    "test_directory": "./climada",
    "test_data": "{test_directory}/test/data",
    "disc_rates": {
        "test_data": "{test_directory}/entity/disc_rates/test/data"
    }
}

Obviously, the default test_directory is given as the relative path to ./climada. This is fine if (but only if) unit or integration tests are started from the installation directory, which is the case in the automated tests on the CI server.
Developers who intend to start a test from another working directory may have to edit this file and replace the relative path with the absolute path to the installation directory:

{
    "_comment": "this is a climada configuration file meant to supersede the default configuration in climada/conf during test",
    "test_directory": "/path/to/installation-dir/climada",
    "test_data": "{test_directory}/test/data",
    "disc_rates": {
        "test_data": "{test_directory}/entity/disc_rates/test/data"
    }
}

Data Initialization#

When import climada is executed in a python script or shell, data files from the installation directory are copied to the location specified in the current configuration.
This happens only when climada is used for the first time with the current configuration. Subsequent execution will only check for presence of files and won’t overwrite existing files.

../_images/FileSystem-2.png

Thus, the home directory will automatically be populated with a climada directory and several files from the repository when climada is used.
To prevent this and keep the home directory clean, create a config file ~/.config/climada.conf with customized values for local_data.system and local_data.demo.
As an example, a file with the following content would suppress creation of directories and copying of files during execution of CLIMADA code:

{
    "local_data": {
        "system": "/path/to/installation-dir/climada/data/system",
        "demo": "/path/to/installation-dir/climada/data/demo"
    }
}
Contents