Use OSM data for RES positioning

[nesnoj]

(preface: this is not issue to be solved in eGo but is related to the developed methods)

There's some room for improvements in the current RES positioning, especially the raster-based methods (WEC, low voltage PV) are inaccurate. Therefore, I'd like to use OSM data to achieve better results. A short description of the current methods can be found in this paper (p 2ff).

Considered types

• MV WEC
• MV PV
• LV PV

OSM data

• OSM tag:key power:generator
• See OSM wiki for source tag specs
• Pitfall: there're point geometries as well as (multi)polygons in OSM (esp. at PV)

Approach
Use OSM data where available and current methods where data is missing or insufficient.

Steps

1. Evaluate OSM coverage (latest OSM dataset) for the above types to see if it's worth the effort
2. Alter the current scripts for RES positioning
3. Validate

Questions

• How can the voltage level of WEC be incorporated? WEC with v_level 4 belong to wind farms which are connected to HV-MV-substation and should not be assigned to sole points but to clusters of points.

PS: The MaStR may be released soon, but the start has been postponed...and postponed...and postponed...so not sure if we'll ever get it.

[Ludee]

How close should this be to the eGoDP REA?
I mean does it use the MV GD and allocates for each of them?
As far as I know there is not much data on LV PV in OSM.

[nesnoj]

Yes, I'd definitely use the MVGDs since every RES requires a grid connection to HV anyway and I think the developed approach is good.
Concerning LV PV: You are right, the focus is on MV WEC+PV. LV PV might not be touched.

Different from the first post's method, a more recent RES data source could be used:

1. netztransparenz.de
   -> initial data, locations are given by addresses :(
2. BNetzA list
   -> updates since new ARegV (2014), locations: UTM coords :)
   Caution: For some reason, the recent version seems to lack the geo positions (the 12/2017 version can be used instead)

So, we have 2 possibilities:

1. Update eGo dataset only using data from dataset 2.
2. Create complete new dataset from datasets 1 and 2 in data processing.

In both cases, OSM points which are not covered by dataset 2 (which holds UTM coord) are used for redistribution of older RES without coords.

Alternatively: The OPSD project is up to release a new dataset based on those data but I neither know the spatial resolution (will they geocode the adresses?) nor if an update is already scheduled.
@wolfbunke, any updates here?

[nesnoj]

@ulfmueller @wolfbunke @IlkaCu
We're planning this RES update for another project. According to my information, there are no further data updates planned in eGo. From your point of view, is it worth to implement this update to work / be in line with the rest of the DP?

[ulfmueller]

We are right now creating a 0.4.4 and 0.4.5 (clean-run of 0.4.4) dp version. Right, that shall be the very last data set generated in the open_eGo project.
Definitely it would be nice if there could be a further dp version in which your changes are included. Very interesting! It would be probably nothing we would have time to evaluate in depth but may be it could be used in other projects.
Do you think it would be too much effort to implement it in-line with the rest of the DP? I just can say, that we unfortunately will not have time to coordinate and/or adjust things in the 'rest of the DP'.

[Ludee]

We will try to make this task compatible with the eGoDP. But it will be separated to not mess anything up ;)

I need an European OSM version anyway, so I will prepare a new upload from Geofabrik. We can choose between the latest monthly (europe-180801.osm.pbf) or the yearly LTS (europe-180101.osm.pbf).

Any wishes?

[ulfmueller]

I like the approach to make it compatible but without messing up stuff which is working and in use!!

I am bit curious, how is your work-flow to separate it? (code-wise and data-wise)
code-wise: creating a new branch within the current data processing? (vs. creating a new repos)
data-wise: creating new tables in existing model_draft and other existing schemata? (vs. creating new database/schemata (probably not))

I somehow think it might be a good a idea to share this work-flow in a transparent way (upfront) to avoid any possible crashes.

[nesnoj]

Good point!
Code: A new branch should suit our needs.
Data: To avoid interferences with current dp runs, we need to create new tables. Same name with an extra prefix or something. Furthermore, all functions that alter other tables which we do not need for testing should be commented out.

@Ludee: We need to know which tables are affected - is the BPMN still valid?
@Ludee: I'd prefer the latest version of OSM.

[nesnoj]

I prefer an update of the eGo data with the BNetzA dataset only. Although we take along errors from the eGo data, I'd avoid a complete re-evaluation of the vast RES data from the TSOs again (OPSD was created to do this).

Concerning licenses of new RES data:

1. BNetzA:

• There's no explicit license given, neither on the page nor in the files.
• Data is published according to MaStRV. The MaStR imprint states, that published data is licensed under "Datenlizenz Deutschland – Namensnennung – Version 2.0". Although they are currently located on the BNetzA website, from my point of view this licence applies (at the latest from MaStR start). So I do not see problems here.

2. Netztransparenz:
   According to the imprint, data must not be published elsewhere (could not find other licensing notes):
   "Inhalt und Gestaltung der Internet-Seiten sind urheberrechtlich geschützt. Eine Vervielfältigung der Seiten oder ihrer Inhalte bedarf der vorherigen schriftlichen Zustimmung per E-Mail der deutschen Übertragungsnetzbetreiber, soweit die Vervielfältigung nicht ohnehin gesetzlich gestattet ist."
   No idea if it is granted by law or not..

[nesnoj]

An updated proposal which incorporates all aspects from above:
(please amend if something is missing)

Updated types

• WEC: type=wind, voltage_level in [4,5]
• PV: type=solar, subtype=ground-mounted, voltage_level in [4,5]
• Biomass: type in [biomass, gas], voltage_level in [4,5]

Current approach, scripts and data

• General approach: See papers 1 (p2 ff) and 2 (p84 f)
• Flow diagram with relevant scripts and tables, see at bottom of diagram (can be opened with e.g. yEd). Caution: Tables refer to release schema model_draft!
• Voltage level allocation schema at eGo Redmine
• RES scripts
• RES versioned at OEP: supply.ego_dp_res_powerplant

Steps
Make sure u use versioned data from v0.4.3 as later versions may still incomplete #318 !

• 1. Prepare and evaluate OSM data
  • Extract points with type and geometry from latest OSM dataset
    • OSM tag:key power:generator (check for more tag combinations)
    • See OSM wiki for source tag specs
    • Pitfall: there're point geometries as well as (multi)polygons in OSM (esp. at PV)
  • Allocate points to grid districts using geom from grid.ego_dp_mv_griddistrict
  • Evaluate OSM coverage for the above types, e.g. create histogram of (OSM point count / RES count) per type
  • Result: Post and discuss coverage in this issue to sort out if it's worth the effort
• 2. Prepare data processing
  • Create new dp branch feature/update_RES_#311 or similar for script amendments.
  • Create new tables for testing in schema 'sandbox', I propose same table names as in dp. Refer to referenced flow diagram above for required tables; only create/copy tables which are updated in RES process (e.g. not demand.ego_dp_loadarea), use original tables from the relevant schemas otherwise.
• 2. Update RES data in versioned RES table. Check with current script of @wolfbunke first.
  • Use BNetzA 12/2017 data from this gist, make sure you carefully read and consider the data notes, translate BNetzA columns to eGo columns. Question: Is the original OPSD table used?
  • Update existing RES, identification may be done using one or more columns of eeg_id, electrical_capacity, start_up_date, generation_type
    • Update capacity and geometry
    • Delete decommissioned
  • Add new RES
  • Allocate RES to grid districts
• 3. Implement new spatial distribution method for distributing in grid districts -> alter the current scripts for RES positioning
  • Use OSM data of where available and current methods where data is missing or insufficient
  • Validate allocation
• 4. Integration [TO BE COMPLETED]
  • ...
• 5. Clean up [TO BE COMPLETED]
  • Add metadata to tables
  • ...

Questions

• Is the update nonrecurring or do we include all steps in DP? Since there's not much time I propose to create a single working dataset only.
• How can the voltage level of WEC be incorporated? WEC with v_level 4 belong to wind farms which are connected to HV-MV-substation and should not be assigned to sole points but to clusters of points.

@Ludee: If you have comments or amendments, please state here asap.

[nesnoj]

Ok, due to available time we agreed on a minimal version:

• No DP integration for now

Part 1

• Test with eGo dataset
• Use OSM points
• Stay with current RES methods
• Update REA scripts to use OSM points for above types as additional allocation points.
  Affected methods:
  • M1, M3, M4
  • exclude M2

Part 2

• Update RES data using ex ante OPSD dataset from @wolfbunke

[Ludee]

Just checked and M1-1 for biogas is completed and data is valid!
Now more methods can be created "quite" easy.
I wish you good luck

[christian-rli]

M3b is complete and produces valid data. The latest commit includes a readme-file that describes setup, M1-1 and M3b. Hope it helps to understand the sql-snippets.

[nesnoj]

Thanks a lot @christian-rli !
I'll have a look on the data.

The next step is to use the new OPSD dataset by @wolfbunke. I expect some scripts from eGo to be reused. I'll get back to you..