Documentation for GCAM
The Global Change Analysis Model
View the Project on GitHub JGCRI/gcam-doc
This page is here to serve as guidance for GCAM developers who may need to update or extend the GCAM Testing Framework. Note the GCAM testing framework is meant to automate many of the steps developers must do to verify their changes do not break GCAM on any of the platforms GCAM is known to run on. The services we use to do this are provided internal to PNNL, as noted below, therefore they cannot be used outside of the network.
Most users will interact with the testing framework only through the pull request interface and will not need to worry about updating the testing framework. Some instances of when they would need to update include:
DESCRIPTION
file)As described in the details below the testing framework is contained in it’s own Git repository but included in the GCAM core repository as a submodule. Thus to get the scripts you will need to update it:
> cd <GCAM workspace>
> git submodule update --init util/testing-framework
To understand how the testing framework is organized and how it all fits together users can read the detailed description below.
The development cycle is the same as any other:
> cd util/testing-framework
> git checkout -b my-new-feature-branch
> # make the required changes
> git add .
> git commit
> # don't forget to push your changes!
> git push
However where it differs is we now need to update the submodule “pointer” in the GCAM workspace as well:
> cd ../../
> git add util/testing-framework
> git commit
> git push
At this point you can open your pull request and the automated tests will use your updated testing framework automatically. In addition no other pull requests will yet be affected! At this point you should also open a pull request in the gcam-testing-framework repository and link to it from your GCAM pull request so all changes can be considered together.
The following software / serices may be useful to review and describe their purpose given users may not be familiar with them. Note we are relying on these services being provided by PNNL’s Dev Central which is currently being provided free of charge code. It does of course imply these tests can only be launched from inside the PNNL network.
We have been using Bitbucket to manage the internal GCAM Git repository. It is only worth mentioning since within the “Pull Request” section is where most users will interact with the testing framework.
Note the JGCRI project is located at: https://stash.pnnl.gov/projects/JGCRI
Jenkins provides a “continuous integration” service, which simply means regularly run some arbitrary set of tests on your version controlled code. It is extremely flexible in terms of how to configure and run tests. In addition PNNL has configured several “build servers” available through Jenkins to use with varying pre-installed tools and operating systems.
Note the JGCRI project is located at: https://ci.pnnl.gov/jenkins/job/JGCRI/
Artifactory is essentially a file server. You can upload any arbitrary files to save there, attach meta data and have multiple versions of. In addition it offers front ends to serve as a local proxy for such services as CRAN (R packages), Docker Hub, PyPi (Python packages), and many more.
You can browse the files at: https://artifactory.pnnl.gov and specific locations where we will save artifacts are noted below.
Not as essential to the testing framework but it will come up so we will describe it. Docker is a set of tools which aims to allow users to package everything required to run their software, including the operating system, inside a minimal client. A user can subsequently “pull” this software container and run it with a docker client on their local operating system, whatever it may be, and can make certain local resource, such as a folder containing GCAM, available inside the docker container. Finally Docker provides a “Docker hub” website as a means to publish your containers, not only for other users to run but also extend to make new containers.
This is a plugin to Bitbucket which allows us to signal Jenkins when to kick off tests. This plug can trigger on any Bitbucket action, such as pull request opened or updated. It makes available a number of useful meta data about what happened and will send it off to some arbitrary URL. In addition it allows us to add a new menu option with in the pull request and even configure arbitrary forms to collect user input to send along too. The full documentation of what is possible is on it’s Github page: https://github.com/tomasbjerre/pull-request-notifier-for-bitbucket
This is the counterpart for the “Pull Request Notifier for Bitbucket”. It will listen for notifications, unpack the metadata, and launch the configured test based off of the metadata. Documentation, again on it’s Github page: https://github.com/jenkinsci/generic-webhook-trigger-plugin
A plugin to update the build status (In Progress, Failure, Success) in Bitbucket. The Jenkins plugin takes care of all of the details for us automatically filling in the name of the test, build number, and link back to Jenkins so a user can watch the progress / look at logs or build artifacts after the fact.
The plugin makes use of Bitbucket Build Status API which actually need to call directly to add / update the builds status of the jobs submitted on PIC.
All of the testing framework scripts and associated data have been placed into their own Git repository at https://stash.pnnl.gov/scm/jgcri/gcam-testing-framework.git
However it has been included in the gcam-core Git repository as a “Submodule” stored in “util/testing-framework”. Again, note that this is on the internal repository so won’t be available form outside the PNNL network. A Git Submodule is essentially a version controlled pointer. It allows us to say version AAA of GCAM requires version BBB of gcam-testing-framework.
Thus, let’s take as an example, if a user creates a branch in which they update the version of the BaseX library. They can create a branch in the testing framework that will install the updated BaseX for testing as well; and update the submodule pointer to point to that branch in the testing framework. In this way the tests from this pull request will run just fine with the updated version of BaseX. And existing pull requests would not be affected so would continue to run just fine with the old version of BaseX.
Most GCAM users will never need to actually “initialize” this submodule and so won’t affect them. Jenkins is configured to “initialize” ALL submodules (which it would have had to do to get hector anyways) so it will get the testing scripts with it’s checkout of the gcam-core.
Jenkins allows us to define tests as arbitrary scripts in what it calls “Jenkinsfile” scripts. This allows us lots of flexibility to do what we need. The downside is the language these scripts are written in is Groovy, a Java based scripting language. Because all of the build steps are contained in these scripts we can re-use them to run multiple Jenkins “tests” such as one triggered by a Bitbucket pull request or one triggered monthly.
We have configured a “Pull Request Notifier” which is triggered on:
RESCOPED_FROM
)It will notify Jenkins with the “JGCRI-gcam-windows” tag with the meta data:
PULL_REQUEST_FROM_BRANCH
PULL_REQUEST_ID
PULL_REQUEST_USER_DISPLAY_NAME
Jenkins will the trigger the JGCRI/gcam-windows test. It will checkout the $PULL_REQUEST_FROM_BRANCH
and run the script util/testing-framework/windows/Jenkinsfile
. It will run on the windows-build-02
build executor which already has the development tools we need in installed:
From there we will need to perform the following steps:
gcamdata
package to generate XMLs
devtools
packagetestthat
packagedevtools::install_deps()
to install the latest version all of the rest of the gcamdata R package dependenciesgcamdata
and run the driver making XMLs onlyJAVA_HOME
so that we can update the PATH
and find jvm.dll
. To do this We replicate the approach we implemented in the run-gcam.bat
script we include in the release package, run: java -cp ..\\libs\\jars\\*;XMLDBDriver.jar XMLDBDriver --print-java-home
gcam.exe -C configuration_ref.xml
database_basexdb
which would otherwise accumulate across all tests over timeWe have configured a “Pull Request Notifier” which is triggered on:
RESCOPED_FROM
)It will notify Jenkins with the “JGCRI-gcam-linux” tag with the meta data:
PULL_REQUEST_FROM_BRANCH
PULL_REQUEST_ID
PULL_REQUEST_USER_DISPLAY_NAME
Jenkins will the trigger the JGCRI/gcam-linux test. It will checkout the $PULL_REQUEST_FROM_BRANCH
and run the script util/testing-framework/linux-docker/Jenkinsfile
. Neither of the linux build executors had the development tools we require already installed, perhaps with the exception of Java. The dev central team recommended using Docker to create the build environment we required, and given the base operating system of your typical Docker image is some flavor of linux this seemed like a good approach. Thus we use the docker-rhel-7
build executor.
Since we want to make sure we are always testing with the very latest R packages we rebuild the rebuild the gcamdata Docker image from scratch each time from util/testing-framework/linux-docker/docker-gcamdata/Dockerfile
.
For the Docker image to compile GCAM we build it from the Dockerfile linux-docker/docker-gcam-build/Dockerfile
.
Then to build and run we will need to perform the following steps:
gcamdata
package to generate XMLs
gcamdata
Docker image as described abovegcam-build
Docker image as described abovegcam-build
docker image run ./gcam.exe -C configuration_ref.xml
database_basexdb
which would otherwise accumulate across all tests over timeWe have configured a Pull Request Notifier “Button” labeled “Launch Validation Runs” and contains the check box form options:
If we should exist delete any previously existing output database per scenario. The default is true so that if a user needs to continually re-run they do not accumulate large databases unnecessarily. It may be useful to uncheck this option however if you made some update to your pull request but need to compare results from the last time it was run to compare behavior.
This defaults to the set of scenario we currently require users to run: Ref, Ref + 2.6; SSPs, SSPs + SPA 2.6. However all of the SPA climate target levels in the committed batch_SSP_SPA*.xml
are available. Note this button form was generated manually but the actual scenarios available are generated from the batch files committed in the gcam-core repo.
When the “button” is clicked and a user confirms, It will notify Jenkins with the “JGCRI-gcam-pic” tag with the meta data:
PULL_REQUEST_FROM_BRANCH
PULL_REQUEST_ID
PULL_REQUEST_USER_DISPLAY_NAME
BUTTON_FORM_DATA
Jenkins will then trigger the JGCRI/gcam-pic test. It will checkout the $PULL_REQUEST_FROM_BRANCH
and run:
$BUTTON_FORM_DATA
into a file user_options.json
which can be processed later when we are setting up the runs on PIC/pic/projects/GCAM/gcam-ci-run/${PULL_REQUEST_ID}
util/testing-framework/pic/run_tests.sh
on PIC via ssh and passing the arguments: $BUILD_NUMBER $PULL_REQUEST_ID $GIT_COMMIT
run_tests.sh
takes care of:
IN_PROGRESS
make xml
cp ~/gcam-core-clean/input/extra/transportation_UCD_SSP* input/extra/
make -j 8 gcam
parse_gcam_batch.R exe/configuration_ref.xml ${FRAMEWORK_DIR}/batch_core_ref.xml
parse_gcam_batch.R exe/configuration_policy.xml ${FRAMEWORK_DIR}/batch_core_policy.xml
parse_gcam_batch.R exe/configuration_ref.xml exe/batch_SSP_REF.xml
parse_gcam_batch.R exe/configuration_ref.xml exe/batch_SSP_SPA1.xml
parse_gcam_batch.R exe/configuration_ref.xml exe/batch_SSP_SPA23.xml
parse_gcam_batch.R exe/configuration_ref.xml exe/batch_SSP_SPA4.xml
parse_gcam_batch.R exe/configuration_ref.xml exe/batch_SSP_SPA5.xml
SUCCESS
parse_gcam_batch.R
takes care of:
xml2
packageuser_options.json
using the jsonlite
packageexpand.grid
to generate permutationspaste0
together FileSet names to generate the scenario nameuser_options[["scenario_run_list"]]
<ScenarioComponents>
<ScenarioComponents>
stage_run.sh
with the arguments: scenario, config_file_name, length(user_options[["clear_output"]]
(the last will give 0 if that check box was not selected and 1 if it was)stage_run.sh
takes care of:
exe_${SCENARIO}
if it existed previouslyexe
to exe_${SCENARIO}
$CONFIGURATION_FILE
to exe_${SCENARIO}/configuration.xml
run_gcam.sbatch
$SCENARIO
run_gcam.sbatch
to the PIC job schedulerOnce run_gcam.sbatch
is run by the SLURM scheduler it:
./gcam.exe -C configuration.xml
stage_run.sh