Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate message_data.model.transport #207

Merged
merged 1,567 commits into from
Aug 1, 2024
Merged
Changes from 1 commit
Commits
Show all changes
1567 commits
Select commit Hold shift + click to select a range
6639eef
Handle model-specific years in .ldv.prepare_computer()
khaeru Feb 6, 2024
3cf4a83
Move common transport keys to a common module
khaeru Feb 6, 2024
9068e73
Add docstrings to some .transport.key
khaeru Feb 6, 2024
400eeec
Test .transport.files using more complete r12 set
khaeru Feb 6, 2024
51fc544
Correct expected dims of ldv-new-capacity.csv
khaeru Feb 6, 2024
859c559
Add "transport other .*" technologies
khaeru Feb 7, 2024
e35fc29
Add transport.test_build.test_debug()
khaeru Feb 7, 2024
0976bac
Add data/transport/r12/energy-other.csv
khaeru Feb 7, 2024
0f2fba9
Specify "other transport" techs explicitly
khaeru Feb 7, 2024
c321d33
Add config for preparing energy-other.csv from iea eweb
khaeru Feb 7, 2024
0395e08
Correct input commodity of "transport other gas"
khaeru Feb 15, 2024
60533b4
Add energy-other.csv to .transport.files
khaeru Feb 15, 2024
6b9967d
Add .transport.non_ldv.other()
khaeru Feb 15, 2024
7362689
Improve output of .transport.test_build.test_debug()
khaeru Feb 15, 2024
9f71c00
Apply updated data for r12_chn per @r-aneeque / iiasa/message_data#538
khaeru Feb 15, 2024
86302aa
Align dims in energy-other.csv, .files, .non_ldv
khaeru Feb 15, 2024
60b79f0
Refresh transport docs
khaeru Feb 15, 2024
6fce642
Transcribe source info from iiasa/message_data#538 to docs
khaeru Feb 15, 2024
8794d97
Adjust data/transport/r12/mode-share/default.csv
khaeru Feb 15, 2024
500ad1b
Adjust data/transport/r12/ldv-activity.csv
khaeru Feb 15, 2024
1a0f391
Use message-ix-models main branch in "transport" ci workflow
khaeru Feb 15, 2024
f585f27
Proof .non_ldv.other() usage against incomplete data
khaeru Feb 15, 2024
06f4aee
Adjust transport tests for iiasa/message_data#535
khaeru Feb 15, 2024
dd478ff
Add coal, fueloil to .transport.data.dummy_supply()
khaeru Feb 15, 2024
a43e9dd
Merge pull request iiasa/message_data#535 from iiasa/transport/2024-w06
khaeru Feb 15, 2024
72614de
Distinguish artifact names in "transport" ci workflow
khaeru Feb 16, 2024
6fc7f01
Add "merge" job to "transport" ci workflow
khaeru Feb 16, 2024
8381e30
Add ethanol to .transport.data.dummy_supply()
khaeru Feb 16, 2024
d321c68
Assign data for t=drail_pub from ikarus source
khaeru Feb 16, 2024
8ba22cb
Add .transport.non_ldv.bound_activity_lo()
khaeru Feb 16, 2024
621f1cc
Give iis when test_build_bare_res() fails to solve
khaeru Feb 16, 2024
49bee45
Use genno.keyseq in .transport.base
khaeru Feb 19, 2024
3f9b0e4
Add minimum activity values to data/transport/r12/config.yaml
khaeru Feb 19, 2024
d96c329
Read minimum activity values to .transport.config
khaeru Feb 19, 2024
2c5316b
Read values from config in .non_ldv.bound_activity_lo
khaeru Feb 19, 2024
3284af7
Temporarily run only ssp2 in "transport" ci workflow
khaeru Feb 19, 2024
30b08f2
Add 5 values for minimum activity in r12
khaeru Feb 20, 2024
1a0797f
Adjust test_get_non_ldv_data
khaeru Feb 20, 2024
da1d819
Hand-tune minimum activity values for r12
khaeru Feb 20, 2024
e292bb3
Add a case to test_build.test_debug()
khaeru Feb 20, 2024
0396dcc
Merge pull request iiasa/message_data#542 from iiasa/transport/2024-w07
khaeru Feb 20, 2024
5e9151d
Bump webfactory/ssh-agent v0.8.0 → v0.9.0
khaeru Feb 22, 2024
fc4a2f2
Merge pull request iiasa/message_data#533 from iiasa/enh/support-pyth…
khaeru Feb 22, 2024
f921e0d
Store a version of r12/energy-other.csv with pipeline included
khaeru Feb 21, 2024
16646a3
Add .transport.operator.where()
khaeru Feb 21, 2024
5ddfb0a
Use "other" as a target name in .groups_iea_eweb()
khaeru Feb 21, 2024
5875540
Use where, not clip, on stage 1 scale factor
khaeru Feb 21, 2024
2a6df18
Set submodules: false for checkout in "transport" ci workflow
khaeru Feb 21, 2024
4b8eaf0
Force upgrade matplotlib in "transport" ci workflow
khaeru Feb 21, 2024
bf35b4c
Add sums to cap:*:non-ldv+units
khaeru Feb 22, 2024
cfd42b3
Place transport plots in url-specific subdir
khaeru Feb 22, 2024
3e424fe
Show scenario url in transport plot subtitle
khaeru Feb 22, 2024
900ce97
Add .transport.plot.energycmdty1
khaeru Feb 22, 2024
a01f3cd
Reenable ssp1 and 3–5 in "transport" ci workflow
khaeru Feb 22, 2024
2e1fa88
Set gams/cplex time limit in "transport" workflow
khaeru Feb 22, 2024
e5101aa
Adjust where() condition for scale-1
khaeru Feb 22, 2024
2644fd6
Update r12/ldv-new-capacity.csv with 2005 values
khaeru Feb 27, 2024
3e420f0
Fix grouping in {fix,var}-cost transport plots
khaeru Feb 27, 2024
2e2658f
Quiet matplotlib logging in .transport.plot
khaeru Feb 27, 2024
0278447
Add cgs from spec, not build info, in add_structure()
khaeru Feb 27, 2024
3287620
Select out:*:ldv and apply units
khaeru Feb 27, 2024
3693812
Refresh "out-share-t{,-cg}-ldv transport plots
khaeru Feb 27, 2024
84167c1
Select historical_new_capacity:*:ldv
khaeru Feb 27, 2024
2952d14
Add cap-new-t-ldv transport plot
khaeru Feb 27, 2024
b68e4e9
Tidy report internals
khaeru Feb 27, 2024
c39e6f4
Refresh demand{,-share} transport plots
khaeru Feb 27, 2024
eedc876
Move {apply,assign}_units to .transport.report
khaeru Feb 27, 2024
0d027f2
Remove transport/report.yaml
khaeru Feb 28, 2024
26feb33
Type hint .transport.operator.make_output_path()
khaeru Feb 28, 2024
e0f83b7
Add reporting labels for all ldv technologies
khaeru Feb 28, 2024
c5fca45
Relax growth_activity_up for ldv technologies
khaeru Feb 28, 2024
51f2039
Use computer class, not str, to type add_iamc_store_write()
khaeru Feb 28, 2024
7fddfad
Simplify path construction in .transport.report
khaeru Feb 28, 2024
75f3861
Allow missing scenario in plot.ggtitle()
khaeru Feb 28, 2024
4a9dd9c
Use "sum" str in .aggregate() to silence pandas warning
khaeru Mar 1, 2024
3ca815c
Add plots across multiple scenarios
khaeru Mar 1, 2024
1c98ceb
Address pandas warning in .transport.operator
khaeru Mar 1, 2024
b40417e
Adjust test_report_bare_solved()
khaeru Mar 1, 2024
7fe598a
Merge pull request iiasa/message_data#543 from iiasa/transport/2024-w08
khaeru Mar 1, 2024
55d3df3
Combine transport gen-activity/debug with "run"
khaeru Mar 5, 2024
e7a4677
Exclude 2020 from "pdt non-active" ssp factor
khaeru Mar 5, 2024
6706c8d
Specify year for .transport.config.fixed_gdp
khaeru Mar 5, 2024
3be3de6
Use keys for pdt per capita check
khaeru Mar 5, 2024
8c3b9ff
Update .transport.cli docstring and docs
khaeru Mar 5, 2024
9575483
Use gha-compatible paths in add_debug()
khaeru Mar 6, 2024
7624f1b
Output processed iea eweb data in add_debug()
khaeru Mar 6, 2024
0476ae4
Simplify add_debug()
khaeru Mar 6, 2024
c33e01f
Also generate debug files when building transport
khaeru Mar 7, 2024
074e884
Update, document share_weight()
khaeru Mar 8, 2024
cd688ec
Use python operators for genno.operator.{div,mul}
khaeru Mar 8, 2024
77e842e
Delete share weight convergence for r12_nam
khaeru Mar 8, 2024
699f7c5
Add plot demand-exo-capita-gdp.pdf
khaeru Mar 8, 2024
ea96d85
Always add all plots from .transport.plot
khaeru Mar 8, 2024
200636c
Include demand-exo-capita-gdp in build debug
khaeru Mar 8, 2024
3829f18
Add gdp-ppp-cap.csv to transport build debug outputs
khaeru Mar 11, 2024
b0febc4
Add .transport.plot.comparepdtcap1
khaeru Mar 11, 2024
4a0dc68
Allow transport build debug w/o base scenario
khaeru Mar 11, 2024
6ea0898
Write transport debug plots in per-scenario directory
khaeru Mar 11, 2024
ca3ddfb
Move .transport.emission tests to parallel module
khaeru Mar 11, 2024
6685e98
Improve ef_for_input()
khaeru Mar 11, 2024
3047e8a
Ignore non-numeric values in sum in build.main()
khaeru Mar 12, 2024
e2bb377
Return an exact set of columns from ef_for_input()
khaeru Mar 12, 2024
671660c
Simplify base_model_data_header() with utils
khaeru Mar 12, 2024
463cc36
Simplify usage of ef_for_input
khaeru Mar 12, 2024
b33b719
Don't write an svg graph in testing.configure_build()
khaeru Mar 12, 2024
d5bf1cd
Improve use of temp platform/scenario in transport workflow
khaeru Mar 12, 2024
f4f394b
Update tests of "debug build" cli usage
khaeru Mar 12, 2024
0913dd9
Apply `ruff format` v0.3.2 to 33 files (whitespace only)
khaeru Mar 12, 2024
7c10671
Shift all ci jobs to self-hosted runner
khaeru Mar 12, 2024
2a4f450
Adjust "pytest" workflow for self-hosted runner
khaeru Mar 12, 2024
9be6d7e
Make _to_csv(…, write_kw=…) optional
khaeru Mar 18, 2024
b6687fd
Consolidate checks for base scenario w/ solution
khaeru Mar 18, 2024
db10bd0
Mark demandexocap1 as a build-stage plot
khaeru Mar 18, 2024
6cc3d8f
Simplify .transport.data.mertoppp
khaeru Mar 18, 2024
5ec6fea
Add .non_ldv.growth_new_capacity() and setting
khaeru Mar 19, 2024
97ad76e
Use non-interactive matplotlib in workflow
khaeru Mar 19, 2024
b04112d
Add initial_new_capacity_up, growth_activity_lo for non-ldvs
khaeru Mar 19, 2024
6866257
Allow "build debug" --from="ssp.* debug build"
khaeru Mar 21, 2024
570cbed
Add (commented) config for debug build
khaeru Mar 21, 2024
6a92be0
Override non-ldv constraints for c=electr techs
khaeru Mar 21, 2024
ab770d5
Report bound_activity_lo-projected.csv
khaeru Mar 21, 2024
62bfa83
Blend scale-1 factor to 1.0 in 2050
khaeru Mar 21, 2024
d3d172a
Adjust typing for genno 1.25.0
khaeru Mar 26, 2024
91b4547
Update .transport.operator.votm() for latest genno
khaeru Mar 25, 2024
1ef95e2
Update test_cli for iiasa/message-ix-models#156
khaeru Mar 26, 2024
e94c616
Adjust expected non-ldv data in tests
khaeru Mar 27, 2024
6ea086f
Temporarily skip 2 transport tests
khaeru Mar 27, 2024
e4a06b5
Merge pull request iiasa/message_data#545 from iiasa/transport/2024-w10
khaeru Mar 27, 2024
77a183b
Use "mix-models fetch" in ci workflows
khaeru Apr 2, 2024
e4eddf9
Use message-ix-models `main` in ci workflows
khaeru Apr 2, 2024
db1ed84
Merge pull request iiasa/message_data#548 from iiasa/fix/ssp-pooch
khaeru Apr 2, 2024
7a87eba
Remove hard-coded structure in dummy_supply()
khaeru Apr 2, 2024
80b7bc6
Remove completed todos in .transport.workflow.generate()
khaeru Apr 2, 2024
71fd0a8
Simplify get_techs()
khaeru Apr 2, 2024
4713a71
Remove skip and xfail marks in transport tests
khaeru Apr 2, 2024
d8cb7ea
Replace "transport build info" context key
khaeru Apr 2, 2024
3347629
Replace "transport spec" context key
khaeru Apr 2, 2024
737f87c
Collect model structure code in .transport.structure
khaeru Apr 2, 2024
ec5d32e
Remove .transport.config.set attribute
khaeru Apr 2, 2024
f2eca84
Adjust use of .config.{set,spec}
khaeru Apr 2, 2024
1444e3a
Update .ldv.usage_data()
khaeru Apr 2, 2024
603562a
Update test_build
khaeru Apr 2, 2024
96bb972
Type hint get_ikarus_data()
khaeru Apr 2, 2024
213ac7c
Use genno.keyseq in .ikarus.prepare_computer()
khaeru Apr 2, 2024
99119f4
Exclude plots in .transport.testing.configure_build
khaeru Apr 2, 2024
68c1420
Sort attributes in .transport.config
khaeru Apr 2, 2024
1cd6472
Format and comment gfei data
khaeru Apr 3, 2024
dbdf979
Remove .tools.gfei_fuel_economy
khaeru Apr 3, 2024
24b3c5e
Allow context.set_scenario() to update url
khaeru Apr 3, 2024
e0d3485
Remove todo in .transport.workflow.generate()
khaeru Apr 3, 2024
6edea03
Remove .projects.shape, .tools.{gea,iea_eei}
khaeru Apr 3, 2024
1c20995
Annotate ldv techs with 'gfei-2017-technology'
khaeru Apr 8, 2024
03ac201
Install dask[dataframe] via message-ix-models[iea-web]
khaeru Apr 10, 2024
ded13fa
Fix use of "transport iamc" keys in "transport all"
khaeru Apr 11, 2024
bb1e785
Inline static data clone in "transport" ci workflow
khaeru Apr 11, 2024
4c1c161
Disable debug quantities in iamc structured report
khaeru Apr 11, 2024
c8561eb
Update r12/ldv-activity.csv per @th-hara calcs
khaeru Apr 16, 2024
e725c89
Update .operator.distance_nonldv() for iiasa/message-ix-models#162
khaeru Apr 16, 2024
9d4f1ef
Add "plot base-fe-intensity-gdp" and file
khaeru Apr 16, 2024
2fa55be
Increase .config.fixed_gdp 300 → 1000 kusd_2005/a
khaeru Apr 18, 2024
815bde9
Xfail gea and shape cases for "debug build" cli
khaeru Apr 18, 2024
ae59a95
Update test_iamc_simulated
khaeru Apr 18, 2024
de20b39
Update docs for iiasa/message_data#549
khaeru Apr 18, 2024
4ed330f
Exclude import from mypy for iiasa/message_data#549
khaeru Apr 19, 2024
fe1c915
Temporarily xfail test_iamc_simulated
khaeru Apr 19, 2024
688eb7b
Merge pull request iiasa/message_data#549 from iiasa/transport/2024-w14
khaeru Apr 20, 2024
d02fe0b
Revise share_weight() handling for ref regions
khaeru Apr 22, 2024
b86239c
Write units header for transport build debug csvs
khaeru Apr 22, 2024
bc9e42d
Skip comments in .plot.read_csvs()
khaeru Apr 23, 2024
1d221d9
Document computations in .demand
khaeru Apr 23, 2024
3b16f0e
Satisfy mypy with pandas-stubs installed
khaeru Apr 23, 2024
5079c81
Compute "calibrate-fe.csv"
khaeru Apr 23, 2024
b29f9db
Rename ldv-input-ref.csv
khaeru Apr 24, 2024
b05ea3b
Construct ldv-input-adj.csv
khaeru Apr 24, 2024
d247e23
Add/use exogenousdatafile.add_tasks()
khaeru Apr 26, 2024
8207e7f
Reference `config` in add_exogenous_data()
khaeru Apr 26, 2024
6beb0dc
Simplify construction of "n:n:ex world"
khaeru Apr 26, 2024
505e4b9
Rename config.fixed_demand → .fixed_pdt
khaeru Apr 26, 2024
d710d71
Add .demand.pdt_per_capita
khaeru Apr 26, 2024
e5b6ed8
Move tests of pdt_per_capita()
khaeru Apr 26, 2024
8647e09
Add .transport.config.policy
khaeru Apr 27, 2024
e79e24f
Add "transport run --base=[auto|bare]" cli option
khaeru Apr 27, 2024
62aa0c6
Add data/transport/base-scenario-url.json
khaeru Apr 27, 2024
6cf1f6c
Improve .transport.workflow
khaeru Apr 27, 2024
1c5f27f
Parametrize "transport" ci workflow on policy setting
khaeru Apr 27, 2024
d193ad5
Remove cli default for --scenario-extra
khaeru Apr 27, 2024
ec247c3
Reverse-transform adjusted gdp/capita
khaeru Apr 29, 2024
48486b8
Use adjusted gdp/capita in mode share calcs
khaeru Apr 29, 2024
0c4aebc
Pin "transport/report" ci jobs to gh-runner-l2
khaeru Apr 29, 2024
ef5fa4d
Update "usage" in transport docs
khaeru Apr 29, 2024
04fdf26
Update transport/doc/input.rst
khaeru Apr 29, 2024
b8b71bd
Pin "transport/report" ci jobs to iiasa-self-hosted-l2
khaeru Apr 29, 2024
a0ec467
Omit both 2020 *and* 2025 from ssp factors
khaeru May 1, 2024
9a36ede
Update doc reference to ldv input files
khaeru May 1, 2024
b063d94
Add data/transport/r12/speed.csv
khaeru May 1, 2024
3409d76
Add "indexers:scenario" via .build.add_structure()
khaeru May 1, 2024
6aee316
Interpolate, select, and use speed from .csv
khaeru May 1, 2024
95cc303
Add (scenario, y) dimensions to load-factor-ldv.csv
khaeru May 1, 2024
65d60a6
Interpolate and select load-factor-ldv.csv
khaeru May 1, 2024
186d362
Apply ldv-input-adj.csv to ldv input intensity
khaeru May 1, 2024
03fd36e
Update share weight convergence in r12/config.yaml
r-aneeque May 17, 2024
66bf77c
Update pdt-elasticity.csv
r-aneeque May 17, 2024
f235e38
Add mode speed differentiation
r-aneeque May 17, 2024
e6b32bc
Update demand.py
r-aneeque May 17, 2024
30a0138
Update transport config.fixed_{gdp,pdt}
r-aneeque May 17, 2024
a6d32fc
Update speed.csv
r-aneeque May 17, 2024
16c2bfd
Update pdt-elasticity.csv
r-aneeque May 17, 2024
ab9dab7
Update ssp base model and policy scenarios for transport
r-aneeque May 21, 2024
d5ebfc3
Remove 1100 and replace with 2100
r-aneeque May 21, 2024
13ad51c
Fix spelling of 'dimensionless'
r-aneeque May 21, 2024
68e7103
Set explicit lpmethod=4, scaind=1 in .transport.workflow
r-aneeque May 22, 2024
1d18231
Update transport base scenarios to ssp(2024) v0.14
r-aneeque Jun 5, 2024
a914ae6
Set growth_activity_up for aviation technologies
khaeru Jun 6, 2024
2c32f42
Extend dynamic constraints for non-ldv techs
khaeru Jun 7, 2024
a2dacaa
Update base year pdt values
r-aneeque Jun 7, 2024
ab24fbe
Tidy formatting in r12/{pdt-elasticity,speed}.csv
khaeru Jul 1, 2024
ab0a2b4
Add placeholder transport data
khaeru Jul 2, 2024
f63a3e1
Add r12/speed.csv data for ssp(2017).2
khaeru Jul 2, 2024
dad2827
Change default transport.config.ssp to ssp(2024).2
khaeru Jul 2, 2024
a66f36f
Adjust transport tests, marks for iiasa/message_data#551
khaeru Jul 2, 2024
fc37472
Merge pull request iiasa/message_data#551 from iiasa/transport/2024-w17
khaeru Jul 2, 2024
c9335ef
Output ue-share.csv from transport
khaeru Jul 2, 2024
cd30b8a
Replace hard-coded 2020 with y₀
khaeru Jul 19, 2024
264f901
Output ue-share-{max,min}.csv from transport
khaeru Jul 19, 2024
9509ba2
Add .transport.operator.{max,min}
khaeru Jul 19, 2024
11119dc
'smooth' transport demand for base model
khaeru Jul 25, 2024
a713c88
Satisfy mypy 1.11 in .transport
khaeru Jul 26, 2024
0cd2675
Merge pull request iiasa/message_data#560 from iiasa/transport/2024-w27
khaeru Jul 26, 2024
dc37a89
Adjust first-party imports in .model.transport
khaeru Jul 29, 2024
a9b2570
Expand branch pattern for "pytest" CI workflow
khaeru Jul 29, 2024
828cd22
Migrate message_data.tools.messagev
khaeru Jul 30, 2024
e094104
Migrate message_data.testing.assert_units
khaeru Jul 30, 2024
bb24906
Adjust first-party imports in .model.transport tests
khaeru Jul 29, 2024
0e232c5
Use transport data/config files from message_ix_models
khaeru Jul 30, 2024
2f96dd3
Migrate pytest markers and options from message_data
khaeru Jul 30, 2024
199599a
Add .transport to .cli.submodules
khaeru Jul 30, 2024
931d3be
Mark tests that fail without non-public data
khaeru Jul 30, 2024
ccbe284
Adjust test marks for #207
khaeru Jul 30, 2024
a6d4fd5
Adjust docs for #207
khaeru Jul 30, 2024
58371b0
Add transport to doc/index
khaeru Jul 29, 2024
4cebc8c
Migrate pyproject.toml entries for .model.transport
khaeru Jul 30, 2024
15282c0
Isolate transport reporting tests
khaeru Jul 30, 2024
dbed4f0
Mark minimum message_ix versions for .transport
khaeru Jul 30, 2024
88a2120
Also xfail on RuntimeError in minimum_version()
khaeru Jul 30, 2024
0ed2260
Type .base.smooth() with Computer class, not str
khaeru Jul 30, 2024
cd2ddce
Add .util.graphviz.HAS_GRAPHVIZ
khaeru Jul 31, 2024
67aabf2
Mark tests that fail w/o graphviz on GHA/macos-13
khaeru Jul 31, 2024
ab0686d
Add #207, #208 to doc/whatsnew
khaeru Jul 29, 2024
f41823a
Merge pull request #208 from iiasa/migrate/transport-adj
khaeru Jul 31, 2024
96c0418
Use GitHub secrets in "transport" CI workflow
khaeru Aug 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add plots across multiple scenarios
khaeru committed Jul 30, 2024

Unverified

This user has not yet uploaded their public signing key.
commit 3ca815c8a959a4d78db7a9482ce0db7d49aad736
229 changes: 218 additions & 11 deletions message_ix_models/model/transport/report.py
Original file line number Diff line number Diff line change
@@ -1,18 +1,21 @@
"""Reporting/postprocessing for MESSAGEix-Transport."""
import logging
from copy import deepcopy
from typing import TYPE_CHECKING
from pathlib import Path
from typing import TYPE_CHECKING, Any, Tuple

from genno import Computer, KeySeq, MissingKeyError
import pandas as pd
from genno import Computer, KeySeq, MissingKeyError, Quantity
from genno.core.key import single_key
from message_ix import Reporter
from message_ix_models import Context
from message_ix_models import Context, ScenarioInfo
from message_ix_models.report.util import add_replacements

from . import Config
from .build import get_spec

if TYPE_CHECKING:
import ixmp
from genno import Computer, Key

log = logging.getLogger(__name__)
@@ -118,26 +121,46 @@ def select_transport_techs(c: "Computer") -> None:
def add_iamc_store_write(c: Computer, base_key) -> "Key":
"""Write `base_key` to CSV, XLSX, and/or both; and/or store on "scenario".
If `base_key` is, for instance, "foo::iamc", this function adds the following keys:
- "foo::iamc+all": both of:
- "foo::iamc+file": both of:
- "foo::iamc+csv": write the data in `base_key` to a file named :file:`foo.csv`.
- "foo::iamc+xlsx": write the data in `base_key` to a file named
:file:`foo.xlsx`.
The files are created in a subdirectory using :func:`make_output_path`—that is,
including a path component given by the scenario URL.
- "foo::iamc+store" store the data in `base_key` as time series data on the
scenario identified by the key "scenario".
.. todo:: Move upstream, to :mod:`message_ix_models`.
"""
# Text fragments: "foo bar" for "foo::bar", and "foo" alone
s, n = str(base_key).replace("::", " "), base_key.name
k = KeySeq(base_key)

file_keys = []
for suffix in ("csv", "xlsx"):
# Create the path
path = c.add(
f"{n} {suffix} path", "make_output_path", "config", name=f"{n}.{suffix}"
k[f"{suffix} path"],
"make_output_path",
"config",
name=f"{k.base.name}.{suffix}",
)
# Write `key` to the path
file_keys.append(c.add(f"{n} {suffix}", "write_report", base_key, path))
file_keys.append(c.add(k[suffix], "write_report", base_key, path))

# Write all files
c.add(f"{s} file", file_keys)
c.add(k["file"], file_keys)

# Store data on "scenario"
c.add(f"{s} store", "store_ts", "scenario", base_key)
c.add(k["store"], "store_ts", "scenario", base_key)

# Both write and store
return single_key(c.add(f"{s} all", [f"{s} file", f"{s} store"]))
return single_key(c.add(k["all"], [k["file"], k["store"]]))


# Units for final energy. This *exact* value (and not e.g. "EJ / year") is required for
@@ -301,7 +324,7 @@ def callback(rep: Reporter, context: Context) -> None:
# Add tasks that prepare data to parametrize the MESSAGEix-GLOBIOM base model
base_key = base.prepare_reporter(rep)

rep.add("transport all", [iamc_key, "transport plots", base_key])
rep.add("transport all", [iamc_key + "file", "transport plots", base_key])

log.info(f"Added {len(rep.graph)-N_keys} keys")
# TODO Write an SVG visualization of reporting calculations
@@ -348,3 +371,187 @@ def configure_legacy_reporting(config: dict) -> None:
group = f"trp {commodity}"
# log.debug(f"{t} → '{group}'")
config[group].append(t.id)


def latest_reporting_from_file(
info: ScenarioInfo, base_dir: Path
) -> Tuple[Any, int, pd.DataFrame]:
"""Locate and retrieve the latest reported output for the scenario `info`.
The file :file:`transport.csv` is sought in a subdirectory of `base_dir` identified
by :attr:`.ScenarioInfo.path`.
.. todo:: Move upstream, to :mod:`message_ix_models`.
Returns
-------
tuple
1. The path of the file read.
2. :class:`int`: The scenario version corresponding to the data read.
3. :class:`pandas.DataFrame`: the data.
If no data is found, all the elements are :any:`None`.
"""
dirs = sorted(base_dir.glob(info.path.replace("vNone", "v*")), reverse=True)
for _dir in dirs:
path = _dir.joinpath("transport.csv")
if not path.exists():
log.info(f"Skip {_dir}; no file 'transport.csv'")
continue
path_version = int(path.parent.name.split("v")[-1])
return (
path,
path_version,
pd.read_csv(path).assign(
Scenario=lambda df: df.Scenario + f"#{path_version}"
),
)

return None, -1, pd.DataFrame()


def latest_reporting_from_platform(
info: ScenarioInfo, platform: "ixmp.Platform", minimum_version: int = -1
) -> Tuple[Any, int, pd.DataFrame]:
"""Retrieve the latest reported output for the scenario described by `info`.
The time series data attached to a scenario on `platform` is retrieved.
.. todo:: Move upstream, to :mod:`message_ix_models`.
Returns
-------
tuple
1. The :class:`.Scenario` object.
2. :class:`int`: The scenario version corresponding to the data read.
3. :class:`pandas.DataFrame`: the data.
If no data is found or the latest version with reporting time series data is
<= `minimum_version`, all the elements are :any:`None`.
"""
from message_ix import Scenario

for _, row in (
platform.scenario_list(model=info.model, scen=info.scenario, default=False)
.sort_values(["version"], ascending=False)
.iterrows()
):
if row.version <= minimum_version:
log.info(f"{row.version} ≤ minimum {minimum_version}")
break
elif row.is_locked:
log.info(f"Skip {info.url} {row.version}; locked")
continue

s = Scenario(
platform, model=info.model, scenario=info.scenario, version=row.version
)
if s.has_solution():
return (
s,
row.version,
s.timeseries().assign(
# Scenario=lambda df: df.Scenario + f"v{row.version}"
),
)
else:
log.info(f"Skip {info.url} {row.version}; no reporting output")
del s

return None, -1, pd.DataFrame()


def multi(context: Context, targets):
"""Report outputs from multiple scenarios."""
import plotnine as p9

report_dir = context.get_local_path("report")
platform = context.get_platform()

dfs = []
for target in map(ScenarioInfo.from_url, targets):
path, path_version, df_path = latest_reporting_from_file(target, report_dir)
scen, scen_version, df_scen = latest_reporting_from_platform(target, platform)

if path_version == scen_version == -1:
raise RuntimeError(f"No reporting output available for {target}")
elif path_version >= scen_version:
source = "file"
df = df_path
version = path_version
else:
source = "platform"
df = df_scen
version = scen_version

log.info(f"{target.url = } {source = } {version = }")

dfs.append(df)

# FIXME This duplicates code from message_ix_models.tools.exo_data; deduplicate
def drop_unique(df, names) -> pd.DataFrame:
names_list = names.split()
for name in names_list:
values = df[name].unique()
if len(values) > 1:
raise RuntimeError(f"Not unique {name!r}: {values}")
return df.drop(names_list, axis=1)

# Convert to a genno.Quantity
cols = ["Variable", "Model", "Scenario", "Region", "Unit"]
data = Quantity(
pd.concat(dfs)
.sort_values(cols)
.melt(id_vars=cols, var_name="y")
.astype({"y": int})
.pipe(drop_unique, "Model")
.rename(columns={"Variable": "v", "Scenario": "s", "Region": "n"})
.dropna(subset=["value"])
.set_index("v s n y Unit".split())["value"]
)

# Select a subset of data
qty = quantity_from_iamc(data, r"Transport\|Stock\|Road\|Passenger\|LDV\|(.*)")

# Plot
# TODO Move to .transport.plot
plot = (
p9.ggplot(qty.to_dataframe().reset_index())
+ p9.aes(x="y", y="value", color="v")
+ p9.facet_grid("n ~ s")
+ p9.geom_point()
+ p9.geom_line()
+ p9.theme(figure_size=(11.7, 16.6))
)
plot.save("debug.pdf")

return data


def quantity_from_iamc(qty: Quantity, variable: str) -> Quantity:
"""Extract data for a single measure from `qty` with (at least) dimensions v, u.
.. todo:: Move upstream, to either :mod:`ixmp` or :mod:`genno`.
Parameters
----------
variable : str
Regular expression to match the ``v`` dimension of `qty`.
"""
import re

from genno.operator import relabel, select

expr = re.compile(variable)
variables, replacements = [], {}
for var in qty.coords["v"].data:
if match := expr.fullmatch(var):
variables.append(match.group(0))
replacements[match.group(0)] = match.group(1)

subset = qty.pipe(select, {"v": variables}).pipe(relabel, {"v": replacements})

unique_units = subset.coords["Unit"].data
assert 1 == len(unique_units)
subset.units = unique_units[0]
return subset.sel(Unit=unique_units[0], drop=True)
6 changes: 6 additions & 0 deletions message_ix_models/model/transport/workflow.py
Original file line number Diff line number Diff line change
@@ -21,6 +21,7 @@ def generate(

from . import build
from .config import Config
from .report import multi

# TODO respect dry-run
# TODO respect quiet
@@ -70,6 +71,7 @@ def generate(
wf.add_step("base", None, target=base_url)

all_keys = []
all_targets = []
for ssp in SSP_2024:
# Construct a label including the SSP
# TODO split to a separate function
@@ -78,6 +80,7 @@ def generate(

# Identify the target of this step
target = url_template.format(label_full)
all_targets.append(target)

# Build Transport on the scenario
# TODO Add functionality like gen-activity
@@ -92,6 +95,9 @@ def generate(
# Report
all_keys.append(wf.add_step(f"{label} reported", f"{label} solved", report))

# Report across multiple scenarios
wf.add("report multi", multi, "context", targets=all_targets)

wf.add("all reported", all_keys)
wf.default_key = "all reported"