Ctrl+K

Continuous Integration and GitHub Actions

Continuous Integration and GitHub Actions#

Automated Tests#

On Jenkins tests are executed and analyzed automatically, in an unbiased environment. The results are stored and can be compared with previous test runs.
Jenkins has a GUI for monitoring individual tests, full test runs and test result trends.
Developers are requested to watch it. At first when they push commits to the code repository, but also later on, when other changes in data or sources may make it necessary to review and refactor code that once passed all tests. The CLIMADA Jenkins server used for continuous integration is at (https://ied-wcr-jenkins.ethz.ch) .

Developer guidelines:#

  • All tests must pass before submitting a pull request.

  • Integration tests don’t run on feature branches in Jenkins, therefore developers are requested to run them locally.

  • After a pull request was accepted and the changes are merged to the develop branch, integration tests may still fail there and have to be addressed.

Test Coverage#

Jenkins also has an interface for exploring code coverage analysis result.
This shows which part of the code has never been run in any test, by module, by function/method and even by single line of code.

Ultimately every single line of code should be tested.

Jenkins Coverage Reports#

To inspect the coverage reports, check out the overview of branch builds on Jenkins. Select the branch or pull request you are interested in. Then, select “Coverage Report” in the menu on the right. Note that this menu entry might not be available if no build of that particular branch/PR succeeded.

You will see a report for every directory and file in CLIMADA. Clicking on a specific file opens a view of the file where the coverage is highlighted.

GitHub Coverage Reports#

To inspect the coverage reports for the GitHub Actions (see below), click on the “Checks” tag in a pull request and then on “GitHub CI” on the left. In the summary of all tasks you will find the “Artifacts” with coverage reports provided as ZIP files. You can download these files, unzip them, and open the resulting HTML files in your browser.

Developer guidelines:#

  • Make sure the coverage of novel code is at 100% before submitting a pull request.

Be aware that having a code coverage alone does not grant that all required tests have been written!
The following artificial example would have a 100% coverage and still obviously misses a test for y(False)

def x(b:bool):
    if b:
        print('been here')
        return 4
    else:
        print('been there')
        return 0

def y(b:bool):
    print('been everywhere')
    return 1/x(b)


import unittest
class TestXY(unittest.TestCase):
    def test_x(self):
        self.assertEqual(x(True), 4)
        self.assertEqual(x(False), 0)

    def test_y(self):
        self.assertEqual(y(True), 0.25)

unittest.TextTestRunner().run(unittest.TestLoader().loadTestsFromTestCase(TestXY));

..

been here
been there
been everywhere
been here

----------------------------------------------------------------------
Ran 2 tests in 0.003s

OK

Static Code Analysis#

At last Jenkins provides an elaborate GUI for pylint findings which is especially useful when working in feature branches.

Observe it!

Developer guidelines:#

  • High Priority Warnings are as severe as test failures and must be addressed at once.

  • Do not introduce new Medium Priority Warnings.

  • Try to avoid introducing Low Priority Warnings, in any case their total number should not increase.

Jenkins Projects Overview#

climada_install_env#

Branch: develop
Runs every day at 1:30AM CET

  • creates conda environment from scratch

  • runs core functionality system test (make install_test)

climada_ci_night#

Branch: develop
Runs when climada_install_env has finished successfully

  • runs all test modules

  • runs static code analysis

climada_branches#

Branch: any
Runs when a commit is pushed to the repository

  • runs all test modules outside of climada.test

  • runs static code analysis

climada_data_api#

Branch: develop
Runs every day at 0:20AM CET

  • tests availability of external data APIs

climada_data_api#

Branch: develop
No automated running

  • tests executability of CLIMADA tutorial notebooks.

GitHub Actions#

CLIMADA has been using a private Jenkins instance for automated testing (Continuous Integration, CI). We recently adopted GitHub Actions for automated unit testing. GitHub Actions is a service provided by GitHub, which lets you configure CI/CD pipelines based on YAML configuration files. GitHub provides servers which ample computational resources to create software environments, install software, test it, and deploy it. See the GitHub Actions Overview for a technical introduction, and the Workflow Syntax for a reference of the pipeline definitions.

The CI results for each pull request can be inspected in the “Checks” tab. For GitHub Actions, users can inspect the logs of every step for every job.

Note#

As of CLIMADA v4.0, the default CI technology remains Jenkins. GitHub Actions CI is currently considered experimental for CLIMADA development.

Unit Testing Guideline#

This pipeline is defined by the .github/workflows/ci.yml file. It contains a single job which will create a CLIMADA environment with Mamba for multiple Python versions, install CLIMADA, run the unit tests, and report the test coverage as well as the simplified test results. The job has a strategy which runs it for multiple times for different Python versions. This way, we make sure that CLIMADA is compatible with all currently supported versions of Python.

The coverage reports in HTML format will be uploaded as job artifacts and can be downloaded as ZIP files. The test results are simple testing summaries that will appear as individual checks/jobs after the respective job completed.

Contents