demeter package

Subpackages

Submodules

demeter.config_reader module

Reads config.ini and type cast parameters.

Copyright (c) 2017, Battelle Memorial Institute

Open source under license BSD 2-Clause - see LICENSE and DISCLAIMER

@author: Chris R. Vernon (chris.vernon@pnnl.gov)

class demeter.config_reader.ReadConfig(params)

Bases: object

DATETIME_FORMAT = '%Y-%m-%d_%Hh%Mm%Ss'
static check_exist(f, kind)

Check file or directory existence.

:param f file or directory full path :param kind either ‘file’ or ‘dir’

:return either path or error

static ck_agg(a)

Check aggregation level. 1 if by only region, 2 if by region and Basin or AEZ.

static ck_ts(t, st_y, ed_y)

Make sure time step fits in year bounds.

Parameters:

  • t – time step

  • st_y – start year

  • ed_y – end year

Returns:

time step

static ck_type(v, p, tp)

Ensure desired type conversion can be achieved.

Parameters:

  • v – value

  • p – name of parameter

Returns:

value

static ck_yr(y, p, lower_year=500, upper_year=3000)

Make sure year is four digits.

Parameters:

  • y – year

  • p – name of parameter

Returns:

int

create_dir(d)

Create directory.

Parameters:

d – Target directory to create

Returns:

Either path or error

get_constraints()

Get a list of constraint files in dir and validate if the user has chosen to use them.

Returns:

List of full path constraint files

get_outdir(out)

Create output directory unique name.

set_target(t)

Set target years to look for when output products. Only the years in this list will be output. If none specified, all will be used.

static vaild_length(value, parameter, max_characters=30)

Ensure len of string is less than or equal to value.

Parameters:

  • value – string

  • parameter – parameter name

  • max_characters – int of max length

Returns:

string

valid_integer(v, p, l)

Ensure target value is an available option.

Parameters:

  • v – value

  • p – name of parameter

  • l – list or tuple of available options for parameter

Returns:

value

valid_limit(v, p, l, typ)

Ensure target value falls within limits.

Parameters:

  • v – value

  • p – name of parameter

  • l – list of start and end range of acceptable values

  • typ – intended type

Returns:

value

static valid_string(v, p, l)

Ensure target value is an available option.

Parameters:

  • v – value

  • p – name of parameter

  • l – list or tuple of available options for parameter

Returns:

value

exception demeter.config_reader.ValidationException(*args)

Bases: Exception

demeter.constraints module

Apply user-defined constraints to base land use layer and predicted allocation data.

Copyright (c) 2017, Battelle Memorial Institute

Open source under license BSD 2-Clause - see LICENSE and DISCLAIMER

@author: Chris R. Vernon (PNNL); Yannick le Page (niquya@gmail.com)

class demeter.constraints.ApplyConstraints(allreg, allaez, final_landclasses, user_years, ixr_ixm, allregaez, spat_region, allregnumber, spat_aez, gcam_landclasses, gcam_regionnumber, gcam_aez, gcam_landname, gcam_array, gcam_ludata, ngrids, constraint_names, observed_landclasses, observed_array, spat_ludata, constraint_files, logger)

Bases: object

apply_constraints_zero()

Apply constraints that occur independent of target year.

Parameters:

self –

Returns:

apply_gcam_constraints(yr_idx, gcam_landmatrix, spat_landmatrix, ixr_ixm_ixg)

Apply user-defined constraints to GCAM land use data.

Returns:

target_change; change per [region, metric, pft]

apply_spat_constraints()

Apply user-defined constraints to base land use layer data. This bins the spatial data into the final functional types.

Returns:

build_gcam_landmatrix()

Create zero array to house GCAM land use summary.

Returns:

Zero 3D array of region, metric, final land classes

build_spat_landmatrix(spat_ludataharm)

Create data summary by region, metric, final land class to conduct the GCAM aggregation.

:return

compile_constraints()

Create array to house constraints. Populate constraints. Position 0 is reserved for kernel density constraints with are created on-the-fly later in the model.

:param constraint_files Array of full path file names to constraint data :param ngrids: Number of grid cells in base land use layer :return: Numpy array housing [ empty (for kernel density), cons_n…]

reg_metric_lcs_zip(lcs_list)

Create list of lists containing [region_ix, metric_ix, land cover class ix].

exception demeter.constraints.ValidationException(*args, **kwargs)

Bases: Exception

demeter.install_supplement module

class demeter.install_supplement.InstallSupplement(example_data_directory)

Bases: object

Download and unpack example data supplement from Zenodo that matches the current installed distribution. :param example_data_directory: Full path to the directory you wish to install

the example data to. Must be write-enabled for the user.

DATA_VERSION_URLS = {'1.3.1': 'https://zenodo.org/record/7240315/files/config_gcam_reference.zip?download=1'}
fetch_zenodo()

Download and unpack the Zenodo example data supplement for the current distribution.

demeter.install_supplement.get_package_data(example_data_directory)

Download and unpack example data supplement from Zenodo that matches the current installed distribution.

Parameters:

example_data_directory (str) – Full path to the directory you wish to install the example data to. Must be write-enabled for the user.

demeter.logger module

Logger

@author Chris R. Vernon @email: chris.vernon@pnnl.gov

License: BSD 2-Clause, see LICENSE and DISCLAIMER files

class demeter.logger.Logger

Bases: object

Initialize project-wide logger. The logger outputs to both stdout and a file.

LOG_FORMAT_STRING = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
static close_logger()

Shutdown logger.

console_handler()

Construct console handler.

file_handler(logfile, write_logfile)

Construct file handler.

demeter.model module

Class to run Demeter model for all defined steps.

Copyright (c) 2017, Battelle Memorial Institute

Open source under license BSD 2-Clause - see LICENSE and DISCLAIMER

@author: Chris R. Vernon (chris.vernon@pnnl.gov)

class demeter.model.Model(**kwargs)

Bases: object

Run the Demeter model.

Parameters:

root_dir – Full path with filename and extension to the directory containing the directory structure.

build_generator()

Construct step generator.

cleanup()

Clean up logger and model instance.

execute()

Execute main downscaling routine.

initialize()

Setup model.

process_step()

Process a single time step.

exception demeter.model.ValidationException(*args)

Bases: Exception

demeter.model.run_model(**kwargs)

Convenience wrapper for the ‘demeter.Model’ class to run all time steps sequentially.

demeter.ncdf_conversion module

class demeter.ncdf_conversion.DemeterToNetcdf(base_year_file: ~typing.Optional[str] = None, start_year: int = 2005, end_year: int = 2005, year_interval: int = 5, xmin: float = -180, xmax: float = 180, ymin: float = -90, ymax: float = 90, resolution: float = 0.05, regrid_resolution: float = 0.05, project_name: str = 'im3', scenario_name: str = '', demeter_version: str = '1.31', csv_input=True, df=Empty DataFrame Columns: [a] Index: [])

Bases: object

Convert Demeter output files to a NetCDF file.

Parameters:

  • base_year_file (str) – Full path with file name and extension to the input base year file.

  • start_year (int) – Start year of the desired extraction.

COMPRESSION_PARAMETERS = {'complevel': 5, 'dtype': 'float32', 'zlib': True}
generate_regrid_coordinates(coord_min: float, coord_max: float, ascending: bool = True, decimals: int = 3) → array

Generate a list of evenly-spaced coordinate pairs for the output grid based on lat, lon values.

Parameters:

  • coord_min (float) – Minimum coordinate in range.

  • coord_max (float) – Maximum coordinate in range.

  • ascending (bool) – Ascend coordinate values if True; descend if False

  • decimals (int) – Number of desired decimals to round to.

Returns:

Array of coordinate values.

generate_scaled_coordinates(coord_min: float, coord_max: float, ascending: bool = True, decimals: int = 3) → array

Generate a list of evenly-spaced coordinate pairs for the output grid based on lat, lon values.

Parameters:

  • coord_min (float) – Minimum coordinate in range.

  • coord_max (float) – Maximum coordinate in range.

  • ascending (bool) – Ascend coordinate values if True; descend if False

  • decimals (int) – Number of desired decimals to round to.

Returns:

Array of coordinate values.

process_output(input_file_directory: str, output_file_directory: str, target_year: int) → Dataset

Process output for a target year for all land classes.

Parameters:

  • input_file_directory (str) – Full path to the directory where demeter stores its outputs.

  • output_file_directory (str) – Full path to the directory where the NetCDF files should be written.

  • target_year (int) – Target year to process.

Returns:

Xarray dataset

demeter.preprocess_data module

class demeter.preprocess_data.FormatGcamDataFrame(df, f_out=None, start_year=2010, through_year=2100, region_name_field='gcam_region_name', region_id_field='gcam_region_id', basin_name_field='glu_name', basin_id_field='basin_id', output_to_csv=False, gcam_landalloc_field='land-allocation', gcam_nodes_field='child-nodes')

Bases: object

Format a data frame output as generated by gcamwrapper for all land categories leaving irrigated and rainfed use as well as management (hi, lo) separated.

Parameters:

  • df (DataFrame) – Pandas dataframe of land allocation from gcamwrapper

  • f_out (str) – Full path with file name and extension to save the output

  • start_year (int) – Start year

  • through_year (int) – Through year

  • region_name_field (str) – Region field name in the GCAM region reference file

  • region_id_field – Region id field name in the GCAM region reference file

  • basin_name_field (str) – Basin GLU abbreviation field name in the GCAM basin reference file

  • basin_id_field (str) – Basin id field name in the GCAM basin reference file

  • output_to_csv (bool) – If True, file will be output to the location specified in the f_out parameter

  • gcam_landalloc_field (str) – Field name for land-allocation in the GCAM database output for land allocation

  • gcam_nodes_field (str) – Field name for gcam nodes field in the GCAM database output for land allocation

Returns:

Data frame; optionally, save as file

Expecting this format from gcamwrapper data frame:

region child-nodes year land-allocation

0 Africa_Eastern Corn_AfrCstE_IRR_hi 1975 0.009381 1 Africa_Eastern Corn_AfrCstE_IRR_hi 1990 0.008705 2 Africa_Eastern Corn_AfrCstE_IRR_hi 2005 0.010365 3 Africa_Eastern Corn_AfrCstE_IRR_hi 2010 0.012348 4 Africa_Eastern Corn_AfrCstE_IRR_hi 2015 0.013194 …

DEMETER_BASIN_NAME_FIELD = 'glu_name'
DEMETER_LANDCLASS_FIELD = 'landclass'
DEMETER_METRIC_FIELD = 'metric_id'
DEMETER_REGID_FIELD = 'region_id'
GCAM_LANDCLASS_DELIM = '_'
GCAM_REGION_FIELD = 'region'
GCAM_UNIT_FIELD = 'units'
GCAM_YEAR_FIELD = 'year'
build_basin_dict()

Create a basin lookup dictionary from the input reference file. File must be comma-separated with fields titled “basin_id” and “glu_name” for the basin id and basin GLU abbreviation, respectively.

Returns:

Dictionary {basin_name: basin_id}

build_regions_dict()

Create a region lookup dictionary from the input reference file. File must be comma-separated with fields titled “region_id” and “region” for the region id and GCAM region name, respectively.

Returns:

Dictionary {region_name: region_id}

format_land_data()

Format land allocation data for use in Demeter. Optional: save to file.

Returns:

Data frame

property gcam_landalloc_field

Field name for land-allocation in the GCAM database output for land allocation

property gcam_nodes_field

Field name for gcam nodes field in the GCAM database output for land allocation

demeter.preprocess_data.format_gcam_data(df, f_out=None, start_year=2010, through_year=2100, region_name_field='gcam_region_name', region_id_field='gcam_region_id', basin_name_field='glu_name', basin_id_field='basin_id', output_to_csv=False, gcam_landalloc_field='land-allocation', gcam_nodes_field='child-nodes')

Convenience wrapper for the FormatGcamDataFrame class. Formats the data frame that gcamwrapper produces into the format the Demeter requires.

Parameters:

  • df (DataFrame) – Pandas dataframe of land allocation from gcamwrapper

  • f_out (str) – Full path with file name and extension to save the output

  • start_year (int) – Start year

  • through_year (int) – Through year

  • region_name_field (str) – Region field name in the GCAM region reference file

  • region_id_field – Region id field name in the GCAM region reference file

  • basin_name_field (str) – Basin GLU abbreviation field name in the GCAM basin reference file

  • basin_id_field (str) – Basin id field name in the GCAM basin reference file

  • output_to_csv (bool) – If True, file will be output to the location specified in the f_out parameter

  • gcam_landalloc_field (str) – Field name for land-allocation in the GCAM database output for land allocation

  • gcam_nodes_field (str) – Field name for gcam nodes field in the GCAM database output for land allocation

Returns:

Data frame; optionally, save as file

demeter.process module

Processes the calculation, map creation, and logging of statistical methods used in determining land use change.

Copyright (c) 2017, Battelle Memorial Institute

Open source under license BSD 2-Clause - see LICENSE and DISCLAIMER

@author: Chris R. Vernon (PNNL); Yannick le Page (niquya@gmail.com)

class demeter.process.ProcessStep(config, s, step_idx, step, write_outputs=False)

Bases: object

Process downscaling of a time step.

expansion_pass()

Conduct expansion pass.

intense_pass(pass_num)

Conduct the first pass of intensification.

outputs()

Create time step specific outputs.

Parameters:

  • spat_ludataharm – harmonized land cover in sqkm per grid cell per land class (n_cells, n_landclasses)

  • spat_ludataharm_orig – same as spat_ludataharm but for the previous time step (n_cells, n_landclasses)

  • cellarea – cell area in sqkm for each grid cell (n_cells)

  • celltrunk – actual percentage of the grid cell included in the data (n_cells)

  • l_fcs – the number of land classes

  • cellindexresin – index of x, y for grid cell location (position, n_cells)

  • lat – geographic coordinate for each latitude in grid

  • lon – geographic coordinate for each longitude in grid

  • step – time step being processed

  • region_coords – full path with extension to region coords file

  • country_coords – full path with extension to country coords file

  • luc_ts_luc – path to luc_timestep output dir

  • transitions – area in sqkm of each transition from one land class to another (n_cells, n_landclasses, n_landclasses)

  • user_years – a list of user selected years to process

prep_step()

Prepare step-specific data.

process()

Process downscaling of a time step.

demeter.reconcile module

The total area in the gridded base layer data may not be equal to the projected data due to water inclusion, or boundary resolution, etc. Therefore, adjustments must be make to the projected data since there is no way of allocating additional land.

Copyright (c) 2017, Battelle Memorial Institute

Open source under license BSD 2-Clause - see LICENSE and DISCLAIMER

@author: Chris R. Vernon (PNNL); Yannick le Page (niquya@gmail.com)

demeter.reconcile.reconcile(allreg, allaez, allregnumber, allregaez, spat_aez, spat_region, spat_ludata, user_years, gcam_ludata, gcam_aez, gcam_regionnumber)

The total area in the gridded base layer data may not be equal to the original GCAM data due to water inclusion, or boundary resolution, etc. Therefore, adjustments must be make to the GCAM data since there is no way of allocating additional land.

Parameters:

  • allreg –

  • allaez –

  • allregnumber –

  • allregaez –

  • spat_aez –

  • spat_region –

  • spat_ludata –

Returns:

demeter.reconcile.reg_metric_zip(allregnumber, allregmetric)

Combine every region index to every contained metric index into a list of lists [region_ix, metric_ix] .

Parameters:

  • allregnumber –

  • allregmetric –

Returns:

demeter.reconcile.yr_reg_metric_zip(yr_list, r_m_list)

Create list of lists containing [yr, region_ix, metric_ix].

Parameters:

  • yr_list –

  • r_m_list –

Returns:

demeter.staging module

Prepare data for processing.

Copyright (c) 2017, Battelle Memorial Institute

Open source under license BSD 2-Clause - see LICENSE and DISCLAIMER

@author: Chris R. Vernon (PNNL); Yannick le Page (niquya@gmail.com)

class demeter.staging.Stage(config)

Bases: object

Prepare data for processing.

harmony()

Harmonize grid area between projected and base layer land allocation.

kernel_filter()

Create kernel density filter.

prep_base()

Prepare base layer land use data.

prep_projected()

Prepare projected land allocation data.

prep_reference()

Read the corresponding reference file to the associated basin or AEZ metric. Also read in region ids.

Parameters:

  • f – Full path with filename and extension to the input file

  • metric – basin or aez

Returns:

Sorted list of metric ids, Sorted list of region ids

set_constraints()

Apply constraints to projected and base layer land allocation and use data.

set_step_zero()

Set data for initial time step

stage()

Run processing that prepares data used in the processing of each time step.

demeter.test_read module

Module contents

Demeter: a land use land cover disaggregation and change detection model.

Copyright (c) 2017, Battelle Memorial Institute

Open source under license BSD 2-Clause - see LICENSE and DISCLAIMER

@author: Chris R. Vernon (PNNL); Yannick le Page (niquya@gmail.com)

class demeter.FormatGcamDataFrame(df, f_out=None, start_year=2010, through_year=2100, region_name_field='gcam_region_name', region_id_field='gcam_region_id', basin_name_field='glu_name', basin_id_field='basin_id', output_to_csv=False, gcam_landalloc_field='land-allocation', gcam_nodes_field='child-nodes')

Bases: object

Format a data frame output as generated by gcamwrapper for all land categories leaving irrigated and rainfed use as well as management (hi, lo) separated.

Parameters:

  • df (DataFrame) – Pandas dataframe of land allocation from gcamwrapper

  • f_out (str) – Full path with file name and extension to save the output

  • start_year (int) – Start year

  • through_year (int) – Through year

  • region_name_field (str) – Region field name in the GCAM region reference file

  • region_id_field – Region id field name in the GCAM region reference file

  • basin_name_field (str) – Basin GLU abbreviation field name in the GCAM basin reference file

  • basin_id_field (str) – Basin id field name in the GCAM basin reference file

  • output_to_csv (bool) – If True, file will be output to the location specified in the f_out parameter

  • gcam_landalloc_field (str) – Field name for land-allocation in the GCAM database output for land allocation

  • gcam_nodes_field (str) – Field name for gcam nodes field in the GCAM database output for land allocation

Returns:

Data frame; optionally, save as file

Expecting this format from gcamwrapper data frame:

region child-nodes year land-allocation

0 Africa_Eastern Corn_AfrCstE_IRR_hi 1975 0.009381 1 Africa_Eastern Corn_AfrCstE_IRR_hi 1990 0.008705 2 Africa_Eastern Corn_AfrCstE_IRR_hi 2005 0.010365 3 Africa_Eastern Corn_AfrCstE_IRR_hi 2010 0.012348 4 Africa_Eastern Corn_AfrCstE_IRR_hi 2015 0.013194 …

DEMETER_BASIN_NAME_FIELD = 'glu_name'
DEMETER_LANDCLASS_FIELD = 'landclass'
DEMETER_METRIC_FIELD = 'metric_id'
DEMETER_REGID_FIELD = 'region_id'
GCAM_LANDCLASS_DELIM = '_'
GCAM_REGION_FIELD = 'region'
GCAM_UNIT_FIELD = 'units'
GCAM_YEAR_FIELD = 'year'
build_basin_dict()

Create a basin lookup dictionary from the input reference file. File must be comma-separated with fields titled “basin_id” and “glu_name” for the basin id and basin GLU abbreviation, respectively.

Returns:

Dictionary {basin_name: basin_id}

build_regions_dict()

Create a region lookup dictionary from the input reference file. File must be comma-separated with fields titled “region_id” and “region” for the region id and GCAM region name, respectively.

Returns:

Dictionary {region_name: region_id}

format_land_data()

Format land allocation data for use in Demeter. Optional: save to file.

Returns:

Data frame

property gcam_landalloc_field

Field name for land-allocation in the GCAM database output for land allocation

property gcam_nodes_field

Field name for gcam nodes field in the GCAM database output for land allocation

class demeter.Model(**kwargs)

Bases: object

Run the Demeter model.

Parameters:

root_dir – Full path with filename and extension to the directory containing the directory structure.

build_generator()

Construct step generator.

cleanup()

Clean up logger and model instance.

execute()

Execute main downscaling routine.

initialize()

Setup model.

process_step()

Process a single time step.

demeter.format_gcam_data(df, f_out=None, start_year=2010, through_year=2100, region_name_field='gcam_region_name', region_id_field='gcam_region_id', basin_name_field='glu_name', basin_id_field='basin_id', output_to_csv=False, gcam_landalloc_field='land-allocation', gcam_nodes_field='child-nodes')

Convenience wrapper for the FormatGcamDataFrame class. Formats the data frame that gcamwrapper produces into the format the Demeter requires.

Parameters:

  • df (DataFrame) – Pandas dataframe of land allocation from gcamwrapper

  • f_out (str) – Full path with file name and extension to save the output

  • start_year (int) – Start year

  • through_year (int) – Through year

  • region_name_field (str) – Region field name in the GCAM region reference file

  • region_id_field – Region id field name in the GCAM region reference file

  • basin_name_field (str) – Basin GLU abbreviation field name in the GCAM basin reference file

  • basin_id_field (str) – Basin id field name in the GCAM basin reference file

  • output_to_csv (bool) – If True, file will be output to the location specified in the f_out parameter

  • gcam_landalloc_field (str) – Field name for land-allocation in the GCAM database output for land allocation

  • gcam_nodes_field (str) – Field name for gcam nodes field in the GCAM database output for land allocation

Returns:

Data frame; optionally, save as file

demeter.get_package_data(example_data_directory)

Download and unpack example data supplement from Zenodo that matches the current installed distribution.

Parameters:

example_data_directory (str) – Full path to the directory you wish to install the example data to. Must be write-enabled for the user.

demeter.run_model(**kwargs)

Convenience wrapper for the ‘demeter.Model’ class to run all time steps sequentially.