Merge pull request #719 from iiasa/enh/macro

Allow mapping distinct names for MESSAGE commodity → MACRO sector

.github/workflows/pytest.yaml

@@ -124,6 +124,11 @@ jobs:
 
     - name: Test documentation build using Sphinx
       if: ${{ startsWith(matrix.os, 'ubuntu') }}
+      env:
+        # For pull_request triggers, GitHub creates a temporary merge commit
+        # with a hash that does not match the head of the branch. Tell it which
+        # branch to use.
+        SPHINXOPTS: -D linkcode_github_remote_head=${{ github.head_ref }}
       run: make --directory=doc html
 
     - name: Upload test coverage to Codecov.io

.gitignore

@@ -23,7 +23,7 @@
 message_ix/model/2*/**
 message_ix/model/$gms*
 message_ix/model/MESSAGE_master.gms
-message_ix/model/cplex.opt
+message_ix/model/cplex.op*
 
 # Sphinx
 doc/_build

RELEASE_NOTES.rst

@@ -5,6 +5,9 @@ All changes
 -----------
 
 - Improve readability of LaTeX equations in docs (:pull:`721`).
+- Bugfix: :meth:`.Scenario.add_macro` would not correctly handle configuration that mapped a MESSAGE (commodity, level) to MACRO sector when the commodity and sector names were different (:pull:`719`).
+- Expand :doc:`macro` documentation, particularly code documentation (:issue:`315`, :pull:`719`).
+- Bugfix: :func:`.computations.as_message_df` would error if a particular dimension was supplied via the `common` argument but not present in `qty` (:pull:`719`).
 
 .. _v3.7.0:
 

doc/api.rst

@@ -91,7 +91,7 @@ Model classes
    GAMSModel
    DEFAULT_CPLEX_OPTIONS
    MESSAGE_ITEMS
-   MACRO_ITEMS
+   ~message_ix.macro.MACRO_ITEMS
 
 .. autodata:: DEFAULT_CPLEX_OPTIONS
 

doc/conf.py

@@ -11,9 +11,9 @@
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 
 try:
-    from importlib.metadata import version
+    from importlib.metadata import version as get_version
 except ImportError:  # Python 3.7
-    from importlib_metadata import version
+    from importlib_metadata import version as get_version
 
 from pathlib import Path
 
@@ -24,7 +24,7 @@
 author = "MESSAGEix Developers"
 
 # The major project version, used as the replacement for |version|.
-version = version("message_ix")
+version = get_version("message_ix")
 # The full project version, used as the replacement for |release| and e.g. in
 # the HTML templates.
 release = version
@@ -46,6 +46,7 @@
     "sphinxcontrib.bibtex",
     "sphinx.ext.autosummary",
     "sphinx.ext.napoleon",
+    "ixmp.utils.sphinx_linkcode_github",
     "message_ix.util.sphinx_gams",
 ]
 
@@ -136,6 +137,10 @@
     "sphinx": ("https://www.sphinx-doc.org/en/master/", None),
 }
 
+# -- Options for sphinx.ext.linkcode / ixmp.utils.sphinx_linkcode_github ---------------
+
+linkcode_github_repo_slug = "iiasa/message_ix"
+
 # -- Options for sphinx.ext.mathjax ----------------------------------------------------
 
 # See https://github.com/iiasa/message_ix/pull/721#pullrequestreview-1497907368:

doc/macro.rst

@@ -1,74 +1,93 @@
 Calibrate and tune MESSAGE-MACRO
 ********************************
 
-Demand elasticities are modelled in MESSAGE via an iterative link to MACRO :cite:`Messner-Schrattenholzer-2000`. 
-The iterative solution process between MESSAGE and MACRO, referred to as "MESSAGE-MACRO", sends information on "energy prices" and "total system costs" from MESSAGE to MACRO, and "demand" and "GDP" from MACRO to MESSAGE, until the demand responses are such that the two models have reached equilibrium (:cite:`Johnson-2016`, further details can be found :ref:`here <message_doc:macro>`). 
-This linkage between the two models is activated by calling :meth:`.solve` with the argument `model='MESSAGE-MACRO'`, or using the GAMS :file:`MESSAGE-MACRO_run.gms` script directly (see :ref:`running` for details about these two methods).
+“MESSAGE-MACRO” refers to an iterative algorithm that links MESSAGE and MACRO :cite:`Messner-Schrattenholzer-2000`.
+This algorithm allows to model demand elasticity: MESSAGE solution data on energy prices and total system costs are provided to MACRO, and MACRO solution data on (endogenous) demand and GDP are provided to MESSAGE.
+This process continues until a convergence criterion or “equilibrium” is reached—briefly, that the MACRO demand output varies minimally between two iterations (:cite:`Johnson-2016`, further details can be found :ref:`here <message_doc:macro>`).
+
+The linked models can be activated by calling :meth:`.Scenario.solve` with the argument `model='MESSAGE-MACRO'`, or using the GAMS :file:`MESSAGE-MACRO_run.gms` script directly (see :ref:`running` for details about these two methods).
 
 .. contents::
    :local:
 
-To solve a scenario in a MESSAGE-MACRO mode, that scenario should be MACRO-calibrated first. As described in :cite:`Johnson-2016`, the calibration process "is parameterized off of a baseline scenario (which assumes some autonomous rate of energy efficiency improvement, AEEI) and is conducted for all MESSAGE regions simultaneously. Therefore, the demand responses motivated by MACRO are meant to represent the additional (compared to the baseline) energy efficiency improvements and conservation that would occur in each region as a result of higher prices for energy services."
+To solve a MESSAGE scenario using MESSAGE-MACRO, it is first necessary to calibrate MACRO.
+As described in :cite:`Johnson-2016`, the calibration process…
+
+   is parameterized off of a baseline scenario (which assumes some autonomous rate of energy efficiency improvement, AEEI) and is conducted for all MESSAGE regions simultaneously.
+   Therefore, the demand responses motivated by MACRO are meant to represent the additional (compared to the baseline) energy efficiency improvements and conservation that would occur in each region as a result of higher prices for energy services.
+
+In the calibration process, the user provides exogenous, reference energy prices (``price_ref``) and reference total energy system cost (``cost_ref``) that correspond to a reference level of demand (``demand_ref``) in a particular **reference year**—generally, the ‘historic’ period that directly precedes the first period in the MESSAGE model horizon for optimization (``firstmodelyear``).
+This reference year is a period for which commodity prices and energy system cost are known for a given demand of those commodities.
 
-In the calibration process, the user defines a reference energy price (`price_ref`) and reference total cost (`cost_ref`) for the energy system that corresponds to a reference value for demand (`demand_ref`) for a base year. Then, using these exogenous reference values plus energy prices (`PRICE_COMMODITY`) and total system cost (`COST_NODAL_NET`) from the solution of MESSAGE for a given demand time series (`demand`), the calibration process changes two parameters, namely, the autonomous rate of energy efficiency improvement (`aeei`) and growth in GDP (`grow`), so that the output of MACRO (`GDP` and `DEMAND`) would converge to an initially specified timeseries trajectory of GDP (`gdp_calibrate`) and demand (`demand`), respectively. The scenario used for calibration is usually a baseline scenario, meaning that this scenario will not include any long-term climate policy targets. Without the calibration, the output of MACRO (`GDP` and `DEMAND`) can be different from the initial exogenous assumptions for GDP and demand (`gdp_calibrate` and `demand`) in MESSAGE for a given scenario.
+Using these reference values plus energy prices (``PRICE_COMMODITY``) and total system cost (``COST_NODAL_NET``) from the solution of MESSAGE for a given demand time series (``demand``), the calibration process changes two parameters, namely, the autonomous rate of energy efficiency improvement (``aeei``) and growth in GDP (``grow``), such that the output of MACRO (``GDP`` and ``DEMAND``) converges to an initially specified time series trajectory of GDP (``gdp_calibrate``) and demand (``demand``), respectively.
+The scenario used for calibration is usually a baseline scenario, meaning that this scenario does not include any constraints that implement policy targets that affect commodity prices (for instance, long-term climate policy targets).
+Without the calibration, the output of MACRO (``GDP`` and ``DEMAND``) can be different from the initial exogenous assumptions for GDP and demand (``gdp_calibrate`` and ``demand``) in MESSAGE for a given scenario.
 
-The calibration process is invoked using the :meth:`.add_macro` on a (baseline) scenario.
-The calibration will be run for the entire optimization-time horizon, i.e., for all model periods after and including, the `firstmodelyear`.
-The calibration process requires some input data, including reference prices, demand, and total system cost of the last historic period in the model, i.e., the model period prior to the `firstmodelyear`, referred to as the "reference year" in the calibration process.
-This "reference year" represents the model period for which commodity prices and energy system cost are known for a given demand of those commodities.
+The calibration process is invoked using :meth:`.Scenario.add_macro` on a (baseline) scenario and runs for the entire optimization-time horizon, i.e., for all model periods including and after the ``firstmodelyear``.
+As mentioned, the required ``price_ref``, ``cost_ref``, and ``demand_ref`` inputs refer to a period prior to the model horizon.
 This is detailed in the :ref:`next section <macro-input-data>`.
 
 The calibration itself is carried out by the :file:`message_ix/model/MACRO/macro_calibration.gms`.
 In this iterative process, `max_it` is used to specify the number of iterations carried out between MESSAGE and MACRO as part of the calibration process.
 The default value is set to 100 iterations, which has proven to be sufficient for the calibration of MACRO to MESSAGE reference scenario for various models.
-Adjustment of GDP growth rates (`grow`) is carried out during even iterations.
-Adjustment of AEEI improvement rates (`aeei`) is carried out during odd iterations.
+Adjustment of GDP growth rates (``grow``) is carried out during even iterations.
+Adjustment of AEEI improvement rates (``aeei``) is carried out during odd iterations.
 
 .. note:: Note, that no actual check is carried out to see if the calibration process has been successful at the end of iterations.
 
 The information from the calibration process is logged in :file:`message_ix/model/MACRO_run.lst`.
 Successful calibration of MESSAGE to MACRO can be identified by looking at the reported values for the "PARAMETER growth_correction" for the last "even" iteration, which should be somewhere around 1e-14 to 1e-16 for positive adjustments or -1e-14 to -1e-16 for negative adjustments.
-Likewise, the "PARAMETER aeei_correction" can be checked for the last "odd" iteration.  
-Once the calibration process has been completed, the scenario will be populated with :ref:`additional parameters <macro-core-formulation>`.
+Likewise, the "PARAMETER aeei_correction" can be checked for the last "odd" iteration.
+Once the calibration process has been completed, the scenario will be populated with :ref:`additional parameters <macro-core>`.
 As part of the calibration process, a final check will automatically be carried out by solving the freshly calibrated scenario in the MESSAGE-MACRO coupled mode, ensuring that the convergence criteria between solution of MESSAGE and MACRO is met after the first iteration.
 
 .. _macro-input-data:
 
-Input data file
-===============
+Input data for calibration
+==========================
 
-The calibration process requires an input data file (Microsoft Excel format), largely built around :ref:`ixmp:excel-data-format`.
-For an example of such input data files, see the files :file:`message_ix/tests/data/*_macro_input.xlsx` included as part of the :mod:`message_ix` test suite; either in your local installation, or `here on GitHub <https://github.com/iiasa/message_ix/tree/main/message_ix/tests/data>`_. The input data file includes the following sheets: 
+For calibration, :meth:`.Scenario.add_macro` requires input data that can be provided as either:
 
-General configuration sheet
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+- a Python :class:`dict` that maps item names to :class:`pandas.DataFrame` objects, or
+- the path to a file in Microsoft Excel format, in which each item is stored as a separate sheet.
 
-  - ``config``: This configuration sheet specifies MACRO-related nodes and years, and maps MACRO sectors to MESSAGE commodities and levels.
-    The sheet has five columns, each of which is a list of labels/codes for a corresponding :ref:`ixmp set <ixmp:data-model-data>`:
-
-    - "node", "year": these can each have any length, depending on the number of regions and years to be included in the MACRO calibration process.
-    - "sector", "commodity", "level": these 3 columns must have equal lengths.
-      They describe a one-to-one mapping between MACRO sectors (entries in the "sector" column) and MESSAGE commodities and levels (paired entries in the "commodity" and "level" columns).
+  For an example of such input data files, see the files :file:`message_ix/tests/data/*_macro_input.xlsx` included as part of the :mod:`message_ix` test suite; either in your local installation, or `here on GitHub <https://github.com/iiasa/message_ix/tree/main/message_ix/tests/data>`_.
 
-MACRO parameter sheets
-~~~~~~~~~~~~~~~~~~~~~~
-   The remaining sheets each contain data for one MACRO parameter:
-
-   - ``price_ref``: prices of MESSAGE commodities in a reference year. 
-     These can be obtained from the variable `PRICE_COMMODITY`. 
-   - ``cost_ref``: total cost of the energy system in the reference year.
-     These can be obtained from the variable `COST_NODAL_NET` and should be divided by a factor of 1000. 
-   - ``demand_ref``: demand for different commodities in the reference year.
-   - ``lotol``: tolerance factor for lower bounds on MACRO variabales.
-   - ``esub``: elasticity between capital-labor and energy.
-   - ``drate``: social discount rate. 
-   - ``depr``: annual percent depreciation.
-   - ``kpvs``: capital value share parameter.
-   - ``kgdp``: initial capital to GDP ratio in base year.
-   - ``gdp_calibrate``: trajectory of GDP in optimization years calibrated to energy demand to MESSAGE.
-     Values for atleast two periods prior to the `firstmodelyear` are required in order to compute the growth rates in historical years. 
-   - ``aeei``: annual potential decrease of energy intensity in sector sector.
-   - ``MERtoPPP``: conversion factor of GDP from market exchange rates to purchasing power parity.
+This section describes the required contents of each item.
+
+``config``: general configuration
+---------------------------------
+
+This table/sheet provides structural information for MACRO calibration and the MESSAGE-MACRO linkage.
+The table has five columns, each of which is a list of labels/codes for a corresponding :ref:`ixmp set <ixmp:data-model-data>`:
+
+- "node", "year": these columns can each have any length, depending on the number of nodes and periods to be included in the MACRO calibration process.
+- "sector", "commodity", "level": these 3 columns must have equal lengths.
+  They describe a one-to-one correspondence between items in the MACRO ``sector`` set (entries in the "sector" column) and MESSAGE ``commodity`` and ``level`` sets (paired entries in the "commodity" and "level" columns).
+
+MACRO parameters
+----------------
+
+The remaining tables/sheets each contain data for one MACRO parameter.
+The required dimensions or symbol of each item are given in the same notation used in the documentation of the :ref:`MACRO core formulation <macro-core>`.
+
+- ``price_ref`` (:math:`n, s`): prices of MACRO sector output in the reference year.
+  These can be constructed from the MESSAGE variable ``PRICE_COMMODITY``, using the ``config`` mapping.
+  If not provided, :mod:`message_ix.macro` will identify the reference year and extrapolate reference values using an exponential function fitted to ``PRICE_COMMODITY`` values; see :func:`.macro.extrapolate`.
+- ``cost_ref`` (:math:`n`): total cost of the energy system in the reference year.
+  These can be constructed from the MESSAGE variable ``COST_NODAL_NET``, including dividing by a factor of 1000.
+  If not provided, :mod:`message_ix.macro` will extrapolate using :func:`.macro.extrapolate`.
+- ``demand_ref`` (:math:`n, s`): demand for MACRO sector output in the reference year.
+- ``lotol`` (:math:`n`): tolerance factor for lower bounds on MACRO variables.
+- ``esub`` (:math:`\epsilon_n`): elasticity of substitution between capital-labor and energy.
+- ``drate`` (:math:`n`): social discount rate.
+- ``depr`` (:math:`\mathrm{depr}_n`): annual percent depreciation.
+- ``kpvs`` (:math:`\alpha_n`): capital value share parameter.
+- ``kgdp`` (:math:`n`): initial capital to GDP ratio in base year.
+- ``gdp_calibrate`` (:math:`n, y`): trajectory of GDP in optimization years calibrated to energy demand to MESSAGE.
+  In order to compute the growth rates in historical years, values are **required** for the reference year *and* one prior period—that is, at least two periods prior to the ``firstmodelyear``.
+- ``aeei`` (:math:`n, s, y`): annual potential decrease of energy intensity in sector sector.
+- ``MERtoPPP`` (:math:`n, y`): conversion factor of GDP from market exchange rates to purchasing power parity.
 
 Numerical issues
 ================
@@ -206,11 +225,38 @@ The arguments can be passed with the solve command, e.g. `scenario.solve(solve_o
 Alternatively the arguments can be specified either in :file:`models.py`.
 
 
-:mod:`message_ix.macro` internals
-=================================
+Code documentation
+==================
 
 .. currentmodule:: message_ix.macro
 
 .. automodule:: message_ix.macro
    :members:
    :exclude-members: MACRO_ITEMS
+
+   The functions :func:`add_model_data` and :func:`calibrate` are used by :meth:`.Scenario.add_macro`.
+   Others are internal; :func:`prepare_computer` assembles the following functions into a :class:`.genno.Computer` that then executes the necessary calculations to prepare the model data.
+
+   .. autosummary::
+      Structures
+      aconst
+      add_par
+      add_structure
+      bconst
+      demand
+      gdp0
+      growth
+      macro_periods
+      mapping_macro_sector
+      price
+      rho
+      total_cost
+      unique_set
+      validate_transform
+      ym1
+
+   The following diagram visualizes the calculation flow:
+
+   .. image:: /_static/macro-calibrate.svg
+      :alt: Diagram of the calculation flow in the calibration of MACRO.
+      :target: ./_static/macro-calibrate.svg

message_ix/core.py

@@ -1,4 +1,5 @@
 import logging
+import os
 from collections.abc import Mapping
 from functools import lru_cache
 from itertools import chain, product
@@ -675,15 +676,22 @@ def solve(self, model="MESSAGE", solve_options={}, **kwargs):
         """
         super().solve(model=model, solve_options=solve_options, **kwargs)
 
-    def add_macro(self, data, scenario=None, check_convergence=True, **kwargs):
+    def add_macro(
+        self,
+        data: Union[Mapping, os.PathLike],
+        scenario=None,
+        check_convergence=True,
+        **kwargs,
+    ):
         """
         Add MACRO parametrization to the Scenario and calibrate.
         Notice: existing MACRO calibration data will be overwritten by running this.
 
         Parameters
         ----------
-        data : dict (str -> DataFrame or Series)
-            Dictionary of required data for MACRO calibration.
+        data : dict (str -> pandas.DataFrame) or path-like.
+            Dictionary of required data for MACRO calibration or path to a file
+            containing the data.
         scenario : string, optional, default: None.
             Scenario name for calibrated MESSAGEix scenario.
         check_convergence : bool, optional, default: True.