AGU 2016 Release

Security Issue

There was a security issue in all of the previous versions of PEcAn. All users are highly encouraged to upgrade to the latest version of PEcAn that fixes this issue. Feel free to contact us for more information about this issue and how to check your instance.

New Models

• FATES
• MAAT

Code Clean-up

• Massive rework of almost the entire code base to better adhere to coding standards and making far more explicit use of R namespaces. A large thanks goes to Ben Bond-Lamberty for initiating this!
• Code clean up uncovered (and in some places created) a wide range of small bugs in the code
• A long standing bug in PEcAn's met processing workflow was an inability to change the temporal range of processed data (i.e. subsequent runs at a site needed to be for the same years). This has now been generalizied across the workflow to update the union of current and new years. The workflow also now distinguishes models that can choose dates from a directory of met data versus models that have to be fed a met driver that is the exact years of the run.

Benchmarking

• Fully-functional workflow now exists that can run benchmarks in the PEcAn database through the workflow based just on settings XML.
• This required splitting read.settings in two and having reference runs inserted in from database
• Generation of FACE benchmarks
• Addition of additional metric functions
• TODO: make it easier to create benchmarks and leverage these tools for general visualization and validation

Parameter Data Assimilation (PDA)

• Fixed mismatch in data alignment due to time zone differences
• Saving covariance matrix for running the MCMC longer
• handling the priors better
• PDA with derived variables (as in SA and EA, although I didn't have a case I needed to use it)
• More likelihoods (Gaussian and multiplicative Laplacian) than Heteroskedastic Laplacian (normal was always there but never really functional I think)
• Building emulator on sufficient statistics surface allowing us to update variances (but we don't insert posteriors for variances to the DB yet)
• Ability to handle non-model parameters like bias terms and precision
• Building one emulator per constraining variable

State Data Assimilation (SDA)

• Generalized SDA code to work better across models with a more standardized interface design
• Also generalized SDA's date handling code so that SDA can be run at different temporal frequencies (previously was hard-coded to annual)
• Added ability to calculate autocorrelation of observation data products and inflate observation covariance
• Generalized from multivariate normal model to multivariate Tobit model to handle zero- and range-bound data (e.g. AGB, fractional composition) in a way that is zero-inflated while also accounting for covariances.
• Initial work (incomplete) toward adding ED2 and LPJ-GUESS couplers for read.restart and write.restart.

Misc

• make: Switched PEcAn build to a make based system. This makes rebuilding quicker and also allows R to more automatically handle downloading required packages.
• add site: Added ability to add new sites on the PEcAn map by right-clicking on the map
• RTM PDA: RTM module now has a system in place to do stand-level parameter data assimilation using the ED2 radiative transfer model.

Camp PEON: Assimilation, multi-site, soil params, Maespa, LPJ-GUESS, improvements to web & tutorials

Virtual machine available for download

State Data Assimilation (SDA)

One of the main goals of this release was to re-generalize the EnKF and generalized ensemble filter that @araiho has been using with LINKAGES back to also working with SIPNET and to bring the design one step closer to working with many models. This is working (yeah!), but not yet fully generalized.

The revised SDA tutorial has been migrated to documentation/tutorials/StateAssimilation and the gitbook updated to explain the use of SDA tags in the settings file and the main functions used.

Multisite runs

The basic PEcAn workflow was substantially re-organized to allow PEcAn to more easily run multiple sites at once (see Site Groups below). The core of the design is to split PEcAn’s settings object (what’s read from the XML) into a list of settings objects, one for each run, and applied to existing functions. Both Settings and SettingsList are now R classes. Within the settings file itself, there is now a <runs> block that can include any number of individual <run>. Where appropriate, PEcAn gathers information across runs — for example, the meta-analysis is performed on the set of PFTs extracted across all runs, rather than being done run-by-run. This change is the first step in a large plan to re-organize the PEcAn workflow to be more modular and also support multi-model runs.

Note: Multisite/multimodel run SettingLists cannot yet be generated through the web interface.

Parameter data assimilation

• Generalized to multiple PFTs
• Emulator now supports a range of additional features that can be controlled through the XML settings:
  • multiple chains
  • extending chains (additional samples) starting from their latest state
  • adding additional knots
  • automated burnin detection

The revised PDA tutorial has been migrated to documentation/tutorials/ParameterAssimilation and the PDA vignette contains a more detailed explanation of how to use the new features.

Non-PFT parameter constraint

As an experiment in giving PEcAn control of non-PFT parameters, we've created a 'soil' PFT for SIPNET that contains soil biogeochemical parameters. If successful, we will extend this approach to other models. This change already improves model calibration and validation and has allowed us to assess the relative contribution of plant vs soil parameters to model output uncertainty.

Models

• Maespa has basic functionality (thanks @mdekauwe for all the help!)
• LPJ-GUESS running, but without PEcAn controlling parameters

Drivers

• Fluxnet LaThuile

Web Interface

• PEcAn landing page updated to be links to tools. Avoids confusion when users forget to add the /pecan, /bety, /rstudio, etc. to the web address.
• Site groups has been populated with a number of major research networks: Ameriflux, Fluxnet, Fluxnet 2015 Tier 1 (open), LTER, NEON, ForestGEO, PalEON
• Links to documentation, gitter, and bug reports from all pages
• Additional mouseover text and links to model and variables tables
• Legend in default plot, removed spline from Shiny plot

Camp PalEON

• This is a special release for the PalEON summer course, and comes pre-installed with extra libraries and data for the course

Tutorials

In addition to migrating the PDA and SDA tutorials to RMarkdown, the model-data comparison at the end of Demo 1 has also been migrated to documentation/tutorials/AnalyzeOutput

Benchmarking

While still in alpha development, @bcow contributed a large chunk of code bringing us much closer to automated benchmark execution and a first pass at a Shiny page for benchmark visualization.

Bugfixes

• C_stack bug in run.sensitivity.analysis: fixed on new VM but older users should upgrade gridExtra R package
• Units in figures and reading of model output #792 -- downstream applications/code should check for changes.
• Can skip meta-analysis in pecan settings. May require updating old pecan.xml, particularly for integration tests.
• Interface: met date checking
• Misc: #906 #916 #919 #923 #948 #950 #951

Known Issues

• This version of PEcAn is misconfigured and is using the wrong server for Brown Dog. The server that is used is the development server for Brown Dog and might result in no data being returned. During the Camp PEON the development server will be using the same code as the production server. If you want to use BrownDog after Camp PEON we highly encourage you to sign up with Brown Dog. We will at that point contact you and send you the instructions to modify your VM to use your new account and the production servers.

CMIP5, Shiny, FLUXNET2015, Global Sensitivity

CMIP5:

We’ve add support for running models into the future under climate change scenarios! This release adds support for grabbing GFDL runs for all four RCP scenarios multiple experiments. RCP4.5 is the ‘default’, but Advanced Edit can be used to change the model (CM3, ESM2M, ESM2G), scenario (rcp26, rcp45, rcp60, rcp85), or ensemble member (r1i1p1, r3i1p1, r5i1p1). We are working with Earth System Grid to provide support for other climate models in future PEcAn releases.

Advanced Visualizations:

We’ve added support for R Shiny visualizations in a way that we hope will allow users to easily add new Shiny apps to the catalog of PEcAn visualizations. Simply drop a new app into the pecan/shiny folder and it should go live. We’ve initially populated our Shiny catalog with two visualizations:

workflowPlot:

Similar to our current visualizations, which plots model output timeseries, but now shows output over all years and without a default 1-day time averaging.

global-sensitivity:

Displays results from our new Monte Carlo global sensitivity analysis. Select outputs by model parameter (all model outputs), by output variable (all model parameters), or dig down into individual parameter-by-output relationships.

FLUXNET2015 and Ameriflux LBL:

At the site-level we’ve added support for the new FLUXNET2015 synthesis product, extending PEcAn’s support for flux tower sites globally. For Ameriflux, we’ve also added support for data on the new Lawrence Berkeley server. Past support for data on the ORNL server will also continue so long as that site remains active, but LBL is the preferred server and will contain the most up-do-date data.

Finally, in response to the above changes and feedback from users, we’ve tried to streamline the main PEcAn tutorial. The Parameter Data Assimilation (PDA) tutorial has also been updated to leverage emulator upgrades in the last release.

Known issues:

MAESPA, G’DAY, LPJ-GUESS not yet ready. JULES supported but not installed on VM, need to download and install yourself.

Met processing bugs: code for extending the date range from previous runs has a bug. Current workaround through advanced edit: set new met output directory (See PDA demo). For CRUNCEP and GFDL, workflow gets tripped up when extending to a new model. Currently requires running met process code by hand in R. Bugfixes will be in next release (Aug 12).

In the pipeline:

Next release will focus on PalEON summer course (SDA, tree ring & allometry modules, hopefully alpha support for PalEON met and NEOTOMA database). Also hope to bring Benchmarking up to a functional beta version, add GLDAS & NLDAS met.

Site Groups; Beta: SDA, emulator, PDA, MAESPA, LPJ-GUESS, TRY; Alpha: benchmarking

• Ability to aggregate collections of sites into groupings. This includes both well recognized networks (Amerflux, Fluxnet, LTER, NEON, etc) and user-defined collections of sites.
• Initial support for the MAESPA and LPJ-GUESS models. Neither is fully-operational via the web interface, but both should run with prescribed inputs. Alpha development on CLM, G'DAY, and MAAT also included.
• Generalization of the parameter data assimilation to be able to use a long list of alternative MCMC sampling schemes implemented in the BayesianTools R package (developed by the COST PROFOUND project) as well as a substantial revitalization and improvement of the emulator-based Bayesian calibration scheme.
• Beta release of a generalization of the Ensemble Kalman Filter state data assimilation code to dynamically estimate both states and process error. More general improvements and generalization in multivariate SDA
• Alpha development of benchmarking. These features require additional database schema changes and further development -- look for a working beta in the next release. Key new concepts include a generalization of data loading based on MIME-type and format, a generalization of model-data alignment, and the beginning of a catalog of skill scores and other statistical metrics.
• Code for merging TRY database into local BETY instance. Because TRY is not public, we are not loading it globally into the database but rather providing tools to allow users to easily upload their own TRY data dumps.
• Various bug fixes in meteorology
• Improvements in RTM package supporting stand-level inversion / calibration

VM available from NCSA

PEcAn 1.4.5 : PRELES + CRUNCEP

This release requires an update to the config file in PEcAn web interface. Without this change the web interface will not work. See config.php changes.

Following are the major new features of this release. Besides these new features many bugs have been fixed, see milestone 1.4.5 list of issues or complete list of commits.

New Models

This release now adds support for the PRELES model.

PRELES

The PREdiction of Light use efficiency, Evapotranspiration, and Soil water (PRELES) model was developed by a team at the Natural Resources Institute Finland with the goal to "develop a semi-empirical, intermediate-complexity model of evapotranspiration and its coupling with canopy photosynthesis" (Peltoniemi et al 2015). Calibrated for boreal regions, it uses common meteorological drivers (temperature, pressure, precipitation, photosynthetically active radiation, and CO2) to predict GPP and water balance (soil water, evaporation, and transpiration). It offers comparatively quick analyses, due to its parsimonious code, making it a unique choice among our suite of model options.

New Met Data

This release adds support for CRUNCEP

CRUNCEP

The Climate Research Unit, National Centers for Environmental Prediction (CRUNCEP) Multi-Scale Synthesis and Terrestrial Model Intercomparison Project (MsTMIP) data set provides climatology data that have been standardized and aggregated for use as input to carbon cycle models at a global (0.5-degree resolution) scale. The data were integrated into gridded global collections of meteorological variables (precipitation, air temperature, air specific humidity, pressure, downward longwave radiation, downward shortwave radiation, and wind speed) and has 6-hour temporal resolution between the years 1901-2010. Using PEcAn’s CRUNCEP function, a user can download a dataset containing these variables. A full description of the data can be found here; https://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=1220.

Web Interface

The web interface now allows the user to login using their credentials, event if the flag authorization required is not set. Any run started when logged in will be recorded as belonging to the logged in user. Only users logged in can now delete workflows.

Users can now filter the history page, by their own runs, those that are available locally, succeeded and by textual search (in notes, model, site).

Notes are no longer an advanced feature, but can now be added during the input section.

config.php changes

A major change has been done to the web interface configuration files. Please make sure to update your config.php, without these changes your web interface will not allow you to do any runs!

The variables $hostlist, $qsuboptions and $qsublist have been merged into a single variable called $hostlist. This list has a list of all hosts that are shown in the host pull down list, as well as any options for qsub. This now also allows you to specify a folder on the remote system to use. All runs will use this remote folder! Please see the example.config.php

The following variables are no longer used : $qsuboptions, $qsublist, $ed_soil, $ed_inputs and can be safely removed from you config.php (after fixing the above).

Additional changes

PEcAn now leverages of Travis CI for continuous integration. To use this in your own branches and forks of PEcAn make sure you have merged in the master branch from PEcAn and created an account on Travis CI. Once that is done you should be able to flip the switch for PEcAn and it will start to build automatically anytime you push changes to your branches/fork.

This patch also marks a change to documentation. We have migrated our github wiki to gitbook and will no longer update the wiki on github. It will allow for easier exportation of PecAn documentation (pdf, epub, mobi, web) as well as make it easier to gain feedback from users (inline commenting).

Known Bugs

The NARR met product will not be processed locally from raw global to model/site specific format and properly added to the database. Unless a global NARR met file exists and is added to BETY, selecting NARR as the met driver for your model will create an error. Those wishing to use NARR are strongly encouraged to select the "Brown Dog" option so that the met is processed remotely rather than locally.

Downloads

The latest version of the VM can be downloaded from NCSA

Citations

Peltoniemi, M., Pulkkinen, M., Aurela, M., Pumpanen, J.,Kolari, P., Makela, A.(2015)."A semi-empirical model of boreal-forest gross primary production , evapotranspiration , and soil water — calibration and sensitivity analysis". Boreal Environment Research, 20(2), 151-171.

PEcAn 1.4.4: Remote execution from web interface

The major new feature of this release of PEcAn is the ability to launch workflows from the web interface that then run on remote machines (e.g. clusters, cloud services). For this to work you will need to have ssh access to that remote machine. PEcAn further supports both remote and local servers that use qsub to schedule job execution instead of just running the job, with PEcAn settings allowing the job submission to be tweaked to particular systems. Running PEcAn remote execution does not require a full PEcAn installation on the remote server; details on remote installation and set-up can be found at remote PEcAn setup

The web interface can now also be configured to control user access to visualizing and downloading existing outputs versus running new workflows. This allows you to provide broad access to model outputs and analyses on your server without risk that unauthorized users will burden compute resources with additional runs. User access is controlled using BETY to configure the users that can view and/or run new workflows.

changes to web interface

One new feature (besides the remote run capability) is the ability to leave notes with the workflow. This allows you to record information about the workflow. Right now this is part of the advanced features, but will be moved in the next release to a more prominent place (#679).

You will need to add the following to you config.php for the ability to control who can run a workflow.

# Require username/password, can set min level to 0 so nobody can
# run/delete.
# 4 = viewer
# 3 = creator
# 2 = manager
# 1 = administrator
$min_run_level=2;
$min_delete_level=2;

For the ability to remote launch instances you will need to add the following to your config.php:

# sshTunnel binary
$SSHtunnel=dirname(__FILE__) . DIRECTORY_SEPARATOR . "sshtunnel.sh";

# List of hosts that need qsub
$qsublist=array("geo.bu.edu");

# List of qsub options, this might only be needed for certain systems
# the list should be the server pointing to an array, the second array
# can contain 3 values (qsub, jobid and qstat).
$qsuboptions=array("geo.bu.edu" => 
                    array("qsub"   => "qsub -V -N @NAME@ -o @STDOUT@ -e @STDERR@ -S /bin/bash",
                          "jobid"  => "Your job ([0-9]+) .*",
                          "qstat"  => "qstat -j @JOBID@ || echo DONE",
                          "job.sh" => "module load udunits R/R-3.0.0_gnu-4.4.6",
                          "models" => array("ED2"    => "module load hdf5")),
                    "host2" => 
                    array( "job.sh" => "echo 'Hello Word'"));

Examples of both these settings are provided in the web/config.example.php file.

Database

This release now will have the ability to sync with the database hosted at the University of Wisconsin - Madison. We have also launched a status page for tracking the overall PEcAn network database synchronization: https://pecan2.bu.edu/pecan/status.php

Changes to PEcAn

As always bug fixes.

Updates to RTM package: additional documentation and vignette updates, support for sensor spectral response functions.

Photosynthesis package: added option for TPU limitation and temperature dependence on parameters

Coming down the pipeline for v1.4.5

The current release of PEcAn includes partial (incomplete) code adding additional models (CLM4.5, PRELES, a revised version of LINKAGES), utilities for benchmarking, support for global meteorology products, integration of remotely-sensed data, and additional generalizations in the state & parameter data assimilation schemes.

Downloads

A VM with this version of PEcAn can be downloaded from NCSA

PEcAn 1.4.3: Improved web accessibility, database sync, meteorology processing. New R vignettes.

Web

• Brown Dog support for processing NARR meteorology in the cloud. You can now run any model for any location in North America without having to download and process 1/2 TB of data!
• Ability to run sensitivity, uncertainty, and ensemble analyses 100% through web interface
• Ability to run MCMC model-data fusion through pecan.xml 'advanced edit', rather than R command line
• VM now comes with Brown Dog enabled
• output from web is now writeable (request by @ankurdesai)

Database

• load.bety.sh now uses transactions (#563) for more stable syncs
• TEST at Brookhaven National Lab now part of PEcAn database sync network
• Numerous updates to support new BETY database constraints

Analysis Modules

• New vignettes for the RTM, photosynthesis, allometry, and priors modules
• New documentation/tutorials
• Changed names of output files to be more informative (#534, #507)
• The RTM module can simulate from and perform a Bayesian (Metropolis-Hastings) inversion of PROSPECT versions 4, 5, and 5B and PROSAIL (single canopy layer). The inversion can also accommodate custom priors (so far, only normal and lognormal) and preset constants (i.e. inversions can be performed on all parameters, or only a subset of unknown parameters if some are known). The inversion process has been rewritten in Fortran to facilitate addition of new models. In addition, users can perform a much slower inversion with fully customizable models and priors (passed as function arguments).

Data Modules

• rh2qair now accepts pressure as an argument, rewritten to be consistent with qair2rh (#518, #528)
• fixes to NARR & PalEON conversions
• support for FACE MIP meteorology

Models

• DALEC: greatly expanded set of supported parameters, priors
• LINKAGES: updates to support code development
• ED2:
  • update ED2IN for r46 (#537)
  • update model2netcdf for latest ED2 code on Github
• CLM4.5: Initial stub for interface modules. Working with Dan Ricciuto to get CLM4.5 on the VM for PEcAn 1.4.4

Notes:

• Ubuntu users: OS v >= 14.04 is required to support nco with netcdf4

PEcAn 1.4.2: MET Process and Brown Dog

PEcAn

Run any model at any location in North America! This version allows for the automated conversion and extraction of NARR and Ameriflux met data on both local and remote (HPC) servers. Processed meteorology is now saved to database for later reuse.

This version of PEcAn now can use BrownDog. BrownDog helps with the conversion of data from input files to model specific files using cloud-based web-services.

PEcAn WEB

• allow the user to search/filter Google Map of sites
• ability to use BrownDog if 'use BrownDog' enabled
• data use agreements displayed for Ameriflux and NARR
• ability to run 'bare ground' ED without vegetation initial conditions (PSS/CSS/Site)
• ability to run parameter data assimilation (MCMC) via advanced edit
• 'Conversion' toggles Google Map between sites with a full set of inputs already in the database vs. sites where inputs can be processed automatically using workflows.
• allow to download input files (even if run has failed)

Modules

• RTM module now support Bayesian inversion of PROSPECT leaf-level spectral model
• Allometry module now has a 'predict' function

As always there are also many bugfixes done

PEcAn 1.4.1: Fetch NARR met; add LINKAGES model

PEcAn

Besides bug fixes, this new version of the VM supports fetching the MET data from NARR and converting it to model specific inputs. As well as a new model LINKAGES (https://github.com/araiho/Linkages).

BETY

Bety is now installed in /home/carya/bety and does not require root to upgrade. Users can simply call the following commands as non root to update the server.

git pull
bundle install
rake db:migrate

PEcAn 1.4.0 : Ameriflux Download

Overview

This version supports conversion of met files. The code can now download and convert the met data to CF standard and to model specific format. The met data will along the way be gap filled. This allows any researchers to pick any Ameriflux site and run the SIPNET model on this site.

As always the latest VM can be downloaded from http://isda.ncsa.illinois.edu/download/index.php?project=PEcAn&sort=version

pecan.xml Changes

The met tag can now support a special notation that lets it download and convert the met data. The met tag can have an input="Ameriflux" and output="sipnet" attribute. Currently these are the only supported conversion but more will be added in the next release of PEcAn.

web changes

When running in the web interface you now have the ability to edit the pecan.xml before the software runs. This allows you to change the number of ensemble runs done, as well as any other changes to the pecan.xml.

After the run finishes, you can now download the sipnet clim file used for a particular run. You can now also plot different variables against each other instead of time vs one of hte variables.