Add `where` helper function to enable reproduction of fancier group constraints #698

brynpickering · 2024-10-18T15:37:34Z

Fixes #604
Fixes #679

We use where a lot now. So, if there is another name we can give this helper function, I'm all ears!

Reviewer checklist

Test(s) added to cover contribution
Documentation updated
Changelog updated
Coverage maintained or improved

brynpickering · 2024-10-18T15:51:09Z

tech:

CCGT:
    base_tech: conversion
    conversion_plus_flag: true
    name: CCGT
    carrier_in: Fuel
    carrier_out: [Power,Heat]
    in_group_1: 
      data: True
      index: [Fuel]
      dims: carriers
    out_group_1:  
      data: True
      index: [Power]
      dims: carriers

Constraint:

constraints:
  balance_conversion:
    description: >-
      Recreate v0.6 conversion plus functionality. Override balance_conversion to tie just in_group_1 and out_group_1 together.
    foreach: [nodes, techs, timesteps]
    equations:
      - where: NOT conversion_plus_flag AND NOT include_storage=true
        expression: sum(flow_out_inc_eff, over=carriers) == sum(flow_in_inc_eff, over=carriers)
      - where: conversion_plus_flag AND NOT include_storage=true
        expression: sum(where(flow_in_inc_eff, in_group_1), over=carriers) == sum(where(flow_out_inc_eff, out_group_1), over=carriers)

codecov · 2024-10-18T16:04:00Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.98%. Comparing base (ed25596) to head (f706ebf).

Additional details and impacted files

@@           Coverage Diff            @@
##             main     #698    +/-   ##
========================================
  Coverage   95.97%   95.98%            
========================================
  Files          29       29            
  Lines        4048     4057     +9     
  Branches      849      579   -270     
========================================
+ Hits         3885     3894     +9     
  Misses         72       72            
  Partials       91       91

Files with missing lines	Coverage Δ
src/calliope/backend/helper_functions.py	`96.98% <100.00%> (+0.12%)`	⬆️

... and 2 files with indirect coverage changes

jmorrisnrel · 2024-10-18T17:08:31Z

This looks great! I've been able to get the default_if_empty alternate approach up and running and have been testing that. I'll keep an eye on the PR and when it's merged we'll test it for our constraints.

sjpfenninger · 2024-10-23T09:55:25Z

We use where a lot now. So, if there is another name we can give this helper function, I'm all ears!

Is it a problem that it has the name of the thing it's doing? I think it's fine, and maybe more confusing if it has a different name.

Perhaps change the signature from where(array, where_array) to where(array, cond) to match the xarray nomenclature and slightly reduce the repetition of where and array?

irm-codebase

The new added functionality is simple enough, but I have some concerns regarding the documentation.
These are small things though, so I can may approve after the reply.

docs/user_defined_math/syntax.md

irm-codebase · 2024-10-23T09:49:42Z

docs/user_defined_math/syntax.md

@@ -215,6 +215,21 @@ But if you're having trouble setting up your math, it is a useful function to ge
    Our internally defined parameters, listed in the `Parameters` section of our [pre-defined base math documentation][base-math] all have default values which propagate to the math.
    You only need to use `default_if_empty` for decision variables and global expressions, and for user-defined parameters.

+### where


Another name for this could be subset or subdim?
It's essentially what is happening.

Not really, as it doesn't need to subset. It has two functions:

potentially applying a condition (may or may not be considered 'subsetting')

potentially extending the dimensions over which the math component is indexed

It can do one or the other, or both.

I wonder if group might work better - it's effectively its purpose as described in #604 - or extend_dims_and_apply_where for maximum clarity & verbosity.

Hmmm... I am not a fan of overly verbose names, and extend_dims_and_apply_where would bloat our syntax too much. The fact that an and is in it maybe hints at it doing too many things.

If this is essentially doing the same thing as the general syntax where, let us keep that name.

irm-codebase · 2024-10-23T09:54:23Z

src/calliope/backend/helper_functions.py

+    """Apply `where` array _within_ an expression string."""
+
+    #:


Seems like an unnecessary line is being 'spammed' to all helper functions. This was probably introduced when more stringent docstring CI was added.

Abstract helper function class, which all helper functions must subclass. The abstract properties and methods defined here must be defined by all helper functions.

Interesting, this is a change. Previously the abstract base class docstring would not filter through to the child classes in docs.

OK, I think I've added an appropriate flag to stop this for helper functions.

irm-codebase · 2024-10-23T10:00:00Z

src/calliope/backend/helper_functions.py

+            >>> flow_cap_max
+            [out] <xarray.DataArray 'flow_cap_max' (nodes: 5, techs: 8)> Size: 320B
+                  array([[   nan, 30000.,    nan,    nan,    nan,    nan,    nan, 10000.],
+                      [   nan,    nan, 10000.,    nan,    nan,    nan,    nan,    nan],
+                      [   nan,    nan, 10000.,    nan,    nan,    nan,    nan,    nan],
+                      [   nan,    nan, 10000.,    nan,    nan,    nan,    nan,    nan],
+                      [ 1000.,    nan,    nan,    nan,    nan,    nan,    nan, 10000.]])
+                  Coordinates:
+                  * nodes    (nodes) object 40B 'region1' 'region1_1' ... 'region1_3' 'region2'
+                  * techs    (techs) object 64B 'battery' 'ccgt' ... 'region1_to_region2'
+            >>> where(flow_cap_max, node_grouping)
+            [out] <xarray.DataArray 'flow_cap_max' (nodes: 5, techs: 8, cap_node_groups: 3)>
+                  array([[[   nan,    nan,    nan],
+                          [30000.,    nan,    nan],
+                          ...
+                          [   nan,    nan,    nan],
+                          [   nan,    nan, 10000.]]])
+                  Coordinates:
+                  * nodes            (nodes) object 40B 'region1' 'region1_1' ... 'region2'
+                  * techs            (techs) object 64B 'battery' ... 'region1_to_region2'
+                  * cap_node_groups  (cap_node_groups) object 24B 'group_1' 'group_2' 'group_3'
+            ```


Perhaps consider removing this code?
Looks like the kind of documentation that may become inadvertently outdated, and the yaml example above is enough to communicate what it does.

irm-codebase

Nice!
I've added some suggestions to the docs, mostly updates for cases referring to the new helper functions page.

irm-codebase · 2024-10-25T08:07:40Z

docs/user_defined_math/helper_functions.md

+For [`where` strings](syntax.md#where-strings) and [`expression` strings](syntax.md#where-strings), there are many helper functions available to use, to allow for more complex operations to be undertaken within the string.
+Their functionality is detailed in the [helper function API page](../reference/api/helper_functions.md).
+Here, we give a brief summary.
+Some of these helper functions require a good understanding of their functionality to apply, so make sure you are comfortable with them before using them.


Suggested change

Some of these helper functions require a good understanding of their functionality to apply, so make sure you are comfortable with them before using them.

Helper functions generally require a good understanding of their functionality, so make sure you are comfortable with them beforehand.

irm-codebase · 2024-10-25T08:08:24Z

docs/user_defined_math/helper_functions.md

+using `inheritance(...)` in a `where` string allows you to grab a subset of technologies / nodes that all share the same [`template`](../creating/templates.md) in the technology's / node's `template` key.
+If a `template` also inherits from another `template` (chained inheritance), you will get all `techs`/`nodes` that are children along that inheritance chain.


Suggested change

using `inheritance(...)` in a `where` string allows you to grab a subset of technologies / nodes that all share the same [`template`](../creating/templates.md) in the technology's / node's `template` key.

If a `template` also inherits from another `template` (chained inheritance), you will get all `techs`/`nodes` that are children along that inheritance chain.

Using `inheritance(...)` in a `where` string allows you to grab a subset of technologies / nodes that all share the same [`template`](../creating/templates.md) in the technology's / node's `template` key.

If a `template` also inherits from another `template` (chained inheritance), you will get all `techs`/`nodes` that are children along that inheritance chain.

irm-codebase · 2024-10-25T08:09:48Z

docs/user_defined_math/helper_functions.md

+Using `sum(..., over=)` in an expression allows you to sum over one or more dimension of your component array (be it a parameter, decision variable, or global expression).
+


Suggested change

Using `sum(..., over=)` in an expression allows you to sum over one or more dimension of your component array (be it a parameter, decision variable, or global expression).

Using `sum(..., over=)` in an expression allows you to sum over one or more dimensions of your component array (be it a parameter, decision variable, or global expression).

irm-codebase · 2024-10-25T08:15:38Z

docs/user_defined_math/syntax.md

+    1.  `any` is a [helper function](helper_functions.md#any); read more below!



Suggested change

1. `any` is a [helper function](helper_functions.md#any); read more below!

1. `any` is a [helper function](helper_functions.md#any)!

irm-codebase · 2024-10-25T08:15:59Z

docs/user_defined_math/syntax.md

+    1.  `get_val_at_index` is a [helper function](helper_functions.md#get_val_at_index); read more below!



Suggested change

1. `get_val_at_index` is a [helper function](helper_functions.md#get_val_at_index); read more below!

1. `get_val_at_index` is a [helper function](helper_functions.md#get_val_at_index)!

irm-codebase · 2024-10-25T08:17:35Z

docs/user_defined_math/syntax.md


 1. Checking the `base_tech` of a technology (`storage`, `supply`, etc.) or its inheritance chain (if using `templates` and the `template` parameter).

    ??? example "Examples"

        - If you want to create a decision variable across only `storage` technologies, you would include `base_tech=storage`.
        - If you want to apply a constraint across only your own `rooftop_supply` technologies (e.g., you have defined `rooftop_supply` in `templates` and your technologies `pv` and `solar_thermal` define `#!yaml template: rooftop_supply`), you would include `inheritance(rooftop_supply)`.
-        Note that `base_tech=...` is a simple check for the given value of `base_tech`, while `inheritance()` is a helper function ([see below](#helper-functions)) which can deal with finding techs/nodes using the same template, e.g. `pv` might inherit the `rooftop_supply` template which in turn might inherit the template `electricity_supply`.
+        Note that `base_tech=...` is a simple check for the given value of `base_tech`, while `inheritance()` is a helper function ([see below](helper_functions.md)) which can deal with finding techs/nodes using the same template, e.g. `pv` might inherit the `rooftop_supply` template which in turn might inherit the template `electricity_supply`.


Suggested change

Note that `base_tech=...` is a simple check for the given value of `base_tech`, while `inheritance()` is a helper function ([see below](helper_functions.md)) which can deal with finding techs/nodes using the same template, e.g. `pv` might inherit the `rooftop_supply` template which in turn might inherit the template `electricity_supply`.

Note that `base_tech=...` is a simple check for the given value of `base_tech`, while `inheritance()` is a [helper function](helper_functions.md) which can deal with finding techs/nodes using the same template, e.g. `pv` might inherit the `rooftop_supply` template which in turn might inherit the template `electricity_supply`.

irm-codebase · 2024-10-25T08:18:08Z

docs/user_defined_math/syntax.md

+    1. `defined` is a [helper function](helper_functions.md#defined); read more below!



Suggested change

1. `defined` is a [helper function](helper_functions.md#defined); read more below!

1. `defined` is a [helper function](helper_functions.md#defined)!

Add helper function, tests, and docs

c4e442e

brynpickering requested review from sjpfenninger and irm-codebase October 18, 2024 15:37

Merge branch 'main' into where-helper-function

1f237ce

Add helper function test to cover conditional

ec3bd32

irm-codebase reviewed Oct 23, 2024

View reviewed changes

Updates following review

f706ebf

brynpickering requested a review from irm-codebase October 24, 2024 11:28

irm-codebase requested changes Oct 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `where` helper function to enable reproduction of fancier group constraints #698

Add `where` helper function to enable reproduction of fancier group constraints #698

brynpickering commented Oct 18, 2024

brynpickering commented Oct 18, 2024

codecov bot commented Oct 18, 2024 •

edited

Loading

jmorrisnrel commented Oct 18, 2024

sjpfenninger commented Oct 23, 2024

irm-codebase left a comment

irm-codebase Oct 23, 2024

brynpickering Oct 24, 2024

irm-codebase Oct 25, 2024

irm-codebase Oct 23, 2024

brynpickering Oct 24, 2024

brynpickering Oct 24, 2024

irm-codebase Oct 23, 2024

irm-codebase left a comment •

edited

Loading

irm-codebase Oct 25, 2024

irm-codebase Oct 25, 2024

irm-codebase Oct 25, 2024

irm-codebase Oct 25, 2024

irm-codebase Oct 25, 2024

irm-codebase Oct 25, 2024

irm-codebase Oct 25, 2024

	Some of these helper functions require a good understanding of their functionality to apply, so make sure you are comfortable with them before using them.
	Helper functions generally require a good understanding of their functionality, so make sure you are comfortable with them beforehand.

		using `inheritance(...)` in a `where` string allows you to grab a subset of technologies / nodes that all share the same [`template`](../creating/templates.md) in the technology's / node's `template` key.
		If a `template` also inherits from another `template` (chained inheritance), you will get all `techs`/`nodes` that are children along that inheritance chain.

		Using `sum(..., over=)` in an expression allows you to sum over one or more dimension of your component array (be it a parameter, decision variable, or global expression).

		1. `any` is a [helper function](helper_functions.md#any); read more below!

		1. `get_val_at_index` is a [helper function](helper_functions.md#get_val_at_index); read more below!

	Note that `base_tech=...` is a simple check for the given value of `base_tech`, while `inheritance()` is a helper function ([see below](helper_functions.md)) which can deal with finding techs/nodes using the same template, e.g. `pv` might inherit the `rooftop_supply` template which in turn might inherit the template `electricity_supply`.
	Note that `base_tech=...` is a simple check for the given value of `base_tech`, while `inheritance()` is a [helper function](helper_functions.md) which can deal with finding techs/nodes using the same template, e.g. `pv` might inherit the `rooftop_supply` template which in turn might inherit the template `electricity_supply`.

		1. `defined` is a [helper function](helper_functions.md#defined); read more below!

Add where helper function to enable reproduction of fancier group constraints #698

Are you sure you want to change the base?

Add where helper function to enable reproduction of fancier group constraints #698

Conversation

brynpickering commented Oct 18, 2024

Reviewer checklist

brynpickering commented Oct 18, 2024

codecov bot commented Oct 18, 2024 • edited Loading

Codecov Report

jmorrisnrel commented Oct 18, 2024

sjpfenninger commented Oct 23, 2024

irm-codebase left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

irm-codebase left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Add `where` helper function to enable reproduction of fancier group constraints #698

Add `where` helper function to enable reproduction of fancier group constraints #698

codecov bot commented Oct 18, 2024 •

edited

Loading

irm-codebase left a comment •

edited

Loading