A workflow management system for GCAM
The pygcam package comprises a set of Python modules and a main driver script designed to facilitate a more efficient workflow using Global Change Assessment Model (GCAM).
What is the “GCAM workflow”?
Using GCAM requires a predictable sequence of steps, including: creating and modifying XML input files for baseline and policy scenarios, running GCAM on all scenarios, running queries to extract results into CSV files, computing differences between each policy scenario and baseline, (often) computing custom results based on generated CSV files, and generating figures. Pygcam automates all of these steps.
What is pygcam?
Pygcam is a Python software framework for managing the GCAM workflow. It is implemented using a layered approach, with each layer designed to expose features important to different users.
GCAM modelers can use the project management framework to define and execute GCAM analyses.
Users with scripting skills can develop custom scripts based on available pygcam plug-ins.
Python programmers can use pygcam as a software library to develop new applications, plug-ins, and other customizations.
Key features of pygcam
XML “setup” language
Rather than manually editing configuration files, the setup system lets you specify in XML the changes required to transform a reference GCAM configuration file to meet your project requirement. Sets of baselines and related policy scenarios can be defined using templates, consolidating and documenting your project setup in one file.
The “GCAM tool” workflow management system
The project XML file describes the steps required to perform your analysis such as which baselines and policy scenarios are available to run, the database queries to extract data into CSV files (with on-the-fly aggregation), charts to generate, and any custom steps (e.g., plug-ins, scripts, or executable programs) you wish.
The “gt” command-line program lets you run it all with one short command, or to specify which scenarios to run or workflow steps to perform.
Built-in support for batch systems such as SLURM and PBS
Project workflow steps can be run locally, or batched for execution on a compute node with the addition of a single command-line flag. Other parameters allow you to specify common batch arguments such as account name, job name, wall time, log file name, and so on. Integrated Monte Carlo support is underway and will be available soon.
Monte Carlo Simulation support
Define parameters, generate trial data, run simulations in Linux clusters, analyze results, interact with results using a web-based tool built into pygcam.
Highly customizable
Write plug-ins to operate as “first class” gt sub-commands, directly callable as project management steps. Write custom Python classes to add functionality to the XML setup system. An extensive configuration system allows you to customize file locations, set default parameters, and choose optional features.
Cross-platform compatibility
The same project definitions can be run under Windows, Mac OS X, and Linux with minor modifications to your pygcam configuration file on each platform.
Open source, free, and documented (really!)
Documentation is available at http://pygcam.readthedocs.io, describing the installation options, configuration system, command-line programs, XML file formats, and programmatic API. A step-by-step tutorial is under development for presentation at the October 2016 GCAM workshop. The full code base is available at https://github.com/JGCRI/pygcam.
Support to help get you started
A limited amount of free support is available to non-commercial users. Commercial consulting support is also available. Contact me (rich at this domain name) for more information.