Thoughts on bigM usage, and configuration defaults

[sstroemer]

This is a cleaned (after discussion with @sjpfenninger on 18.07.) list of things that I noticed while digging into v0.7, given in no particular order. I've created a separate issue (#645) for the calculation of investment costs, since I felt that was too confusing/long to put in here.

About bigM

The parameter bigM is used as a general big M in various places, but also as "penalty cost coefficient" for the unmet_demand (and unmet_supply) slack/feasibility variables. If a user encounters that in min_cost_optimisation, they may be tempted to pick a value that they relate to load shedding, or VoLL, which could lead to silent "mistakes".

The warning, "This should only be considered a debugging rather than a modelling tool as it may distort the model in other ways due to the large impact it has on the objective function value.", is somewhat "hidden" in the unmet_demand (or unmet_supply) documentation, but could be more prominently featured where that penalty is used.

Also note the comment in the national scale example, which may be similarly misleading (and somewhat contradicts the paragraph right below the first mention):

bigM: 1e5  # Sets the scale of unmet demand, which cannot be too high, otherwise the optimisation will not converge

Since the current parameter is - in some places - not really a "big M" in its initial meaning, maybe the introduction of an explicit config.feasibility_penalty (or similar) setting could be useful, and make it clear what the "constraint reformulation big M" is and what the "unmet demand/supply penalty" is.

Note: The current default (1e6) in the urban scale example (line 33 @ model.yaml) actually may even lead to a wrongly infeasible model with the default solver (cbc), if a user uses this config as starting point for small MILP models and some coefficients, e.g., for costs or capacity/flow bounds are small (as in: it's numerically unstable, leading Cbc to an infeasibility, while Gurobi manages to find a feasible solution).

Picking bigM based on assumptions

async_flow_in_milp (and async_flow_out_milp) use a general bigM. Since we know, that flow_in may have an implicit upper bound, given by the upper bound on the corresponding flow_in_max [compare], defined by (timestep_resolution and) the upper bound of flow_cap (= flow_cap_max), a tight bigM could be determined in advance (especially an exact one for technologies without active investment, if flow_cap_min = flow_cap_max). That could (depending on the model, configuration, solver, ...) heavily influence solving times (roughly: the LP relaxation may be extremely bad right now).

A similar approach might be good where ever bigM is used: Figure out - given assumptions X - what a tight bound could be, and if X does not hold fall back to a general, globally defined, value for bigM. I know, unit commitment is currently not fully supported, but proper choice of bigM can have a immense impact there as soon as, e.g., partial load is activated.

Transmission technologies

As I understand, the reason behind symmetric_transmission (and therefore also link_flow_cap) is that the two related transmission links do not explicitly "know" each other, but still need to adhere to the same capacity. Is there any other way with the current math implementation to work around that (e.g., something like the old "lookup_remotes")? These auxiliary variables and constraints may - depending on solution method, ... - be hard to solve.

Wasting energy

async_flow_switch implements a way to prevent simultaneous charging/discharging. Optimal operation of batteries given a not non-negative price time series as input may be a prime example for that. However, the same underlying problem may occur for lossy transmissions, or more generally for any "conversion loop" present in the system. An example: power-to-gas > gas > fuell-cell > electricity > .... These cycles may be extremely hard to determine in a model, so automatic detection and modification may be "impossible".

However, a warning in the documentation could be useful, like "Common problems in models with situations of over-supply" (or similar), explaining the problem and possible solutions (e.g., custom math). Otherwise, a user may set force_async_flow = True, not expecting that a similar problem may occur elsewhere.

Storages - see #650

[sjpfenninger]

Thanks for this! We're going to immediately track the storage issue in #650 and will revisit the other points (which are all valid) later after the initial 0.7 release.