diff --git a/README.rst b/README.rst index 526cc5e6..9b0e11c2 100644 --- a/README.rst +++ b/README.rst @@ -9,7 +9,7 @@ eGo dataprocessing =================== -(geo)data processing, database setup, data validation +See the `documentation `_ hosted on ReadTheDocs. ========= Copyleft @@ -17,78 +17,3 @@ Copyleft Code licensed under "GNU Affero General Public License Version 3 (AGPL-3.0)" It is a collaborative work with several copyright owner: Cite as "eGo dataprocessing © Flensburg University of Applied Sciences, Centre for Sustainable Energy Systems © Europa-Universität Flensburg, Centre for Sustainable Energy Systems © Reiner Lemoine Institut © DLR Institute for Networked Energy Systems" - -Installation -============ - -Installation of latest release - -``` -(sudo) pip3 install dataprocessing -``` - -Installation (of developer version) via pip on linux systems as follows - -``` -sudo pip3 install -e -``` - -Troubleshooting - -On Windows -*********** - -1. __Problem__: Installation of required package shapely fails. __Solution__: Install pre-build - wheel from [here](http://www.lfd.uci.edu/~gohlke/pythonlibs/#shapely) - - -Run -==== - - You can run data processing by calling command-line script - - ``` - ~$ ego_data_processing - ``` - -Local database with docker -========================== - -You need to have `docker-compose` installed. And some more tools, see requirements_system.txt. - -### Spin up database with docker - -``` -docker-compose up -d --build -``` - -### Database credentials - -| database | dp | -| port | 54321 | -| user | oeuser | -| password | egon | - - -Usage -===== - - -Pre-processing -************** - -Execute `eGo_import_and_preprocessing.py` for starting pre-processing from scratch doing data import and a pre-processing of imported tables. - -Once completed, export required schemas to SQL dump - -_You might have to install `postgresql-client-11` from additional source to have a compatible client with the v11 dockered PostgreSQL database._ - -``` - /usr/lib/postgresql/11/bin/pg_dump -d dp -U oeuser -p 54321 -h localhost -Fc -n model_draft -n openstreetmap -n boundaries -n society > `date -I`_eGo_data_pre-processing_bremen.backup -``` - -Import a dump by - -``` -/usr/lib/postgresql/11/bin/pg_restore -d dp -U oeuser -p 54321 -h localhost 2020-03-28_eGo_data_pre-processing_deu.backup -``` diff --git a/documentation/bpmn/ego_dp-pre_bpmn_section_cosmoclmgrid.graphml b/documentation/bpmn/ego_dp-pre_bpmn_section_cosmoclmgrid.graphml index c086fb01..b19fcae9 100644 --- a/documentation/bpmn/ego_dp-pre_bpmn_section_cosmoclmgrid.graphml +++ b/documentation/bpmn/ego_dp-pre_bpmn_section_cosmoclmgrid.graphml @@ -17,11 +17,11 @@ - + - eGo Pre-Processing - cosmoclmgrid - + eGo Pre-Processing - cosmoclmgrid + @@ -53,18 +53,18 @@ - + - + - + @@ -255,7 +255,6 @@ - diff --git a/documentation/bpmn/ego_dp-pre_bpmn_section_demand_federalstate.graphml b/documentation/bpmn/ego_dp-pre_bpmn_section_demand_federalstate.graphml index b48a5076..558d0903 100644 --- a/documentation/bpmn/ego_dp-pre_bpmn_section_demand_federalstate.graphml +++ b/documentation/bpmn/ego_dp-pre_bpmn_section_demand_federalstate.graphml @@ -17,11 +17,11 @@ - + - eGo Pre-Processing - demand federal states - + eGo Pre-Processing - demand federal states + @@ -53,7 +53,7 @@ - + @@ -64,7 +64,7 @@ - + diff --git a/documentation/bpmn/ego_dp-pre_bpmn_section_osmtgmod.graphml b/documentation/bpmn/ego_dp-pre_bpmn_section_osmtgmod.graphml index c877738f..c65ff1e1 100644 --- a/documentation/bpmn/ego_dp-pre_bpmn_section_osmtgmod.graphml +++ b/documentation/bpmn/ego_dp-pre_bpmn_section_osmtgmod.graphml @@ -17,11 +17,11 @@ - + - eGo Pre-Processing - osmTGmod - + eGo Pre-Processing - osmTGmod + @@ -31,40 +31,40 @@ + + + - - - - - + + - + - + - + @@ -167,10 +167,10 @@ grid-otg_ehvhv_results_metadataOSM Import + - @@ -219,7 +219,6 @@ grid-otg_ehvhv_results_metadata - diff --git a/documentation/bpmn/ego_dp-pre_bpmn_section_scenario-log.graphml b/documentation/bpmn/ego_dp-pre_bpmn_section_scenario-log.graphml index ea5d5619..7dfc07bb 100644 --- a/documentation/bpmn/ego_dp-pre_bpmn_section_scenario-log.graphml +++ b/documentation/bpmn/ego_dp-pre_bpmn_section_scenario-log.graphml @@ -17,7 +17,7 @@ - + eGo Pre-Processing - scenario_log @@ -53,13 +53,13 @@ - + - + diff --git a/documentation/bpmn/ego_dp-pre_bpmn_section_slp.graphml b/documentation/bpmn/ego_dp-pre_bpmn_section_slp.graphml index 2504ad6a..b272963d 100644 --- a/documentation/bpmn/ego_dp-pre_bpmn_section_slp.graphml +++ b/documentation/bpmn/ego_dp-pre_bpmn_section_slp.graphml @@ -17,11 +17,11 @@ - + - eGo Pre-Processing - slp - + eGo Pre-Processing - slp + @@ -53,18 +53,18 @@ - + - + - + @@ -227,7 +227,6 @@ - diff --git a/documentation/bpmn/ego_dp-pre_bpmn_section_slp_parameters.graphml b/documentation/bpmn/ego_dp-pre_bpmn_section_slp_parameters.graphml index 362b8fc6..7ba2a5ac 100644 --- a/documentation/bpmn/ego_dp-pre_bpmn_section_slp_parameters.graphml +++ b/documentation/bpmn/ego_dp-pre_bpmn_section_slp_parameters.graphml @@ -17,11 +17,11 @@ - + - eGo Pre-Processing - slp_parameters - + eGo Pre-Processing - slp_parameters + @@ -31,40 +31,40 @@ + + + - - - - - + + - + - + - + @@ -256,7 +256,6 @@ - diff --git a/documentation/bpmn/ego_dp-pre_bpmn_section_supply.graphml b/documentation/bpmn/ego_dp-pre_bpmn_section_supply.graphml index 3af490e1..839d0625 100644 --- a/documentation/bpmn/ego_dp-pre_bpmn_section_supply.graphml +++ b/documentation/bpmn/ego_dp-pre_bpmn_section_supply.graphml @@ -17,11 +17,11 @@ - + - eGo Pre-Processing - supply - + eGo Pre-Processing - supply + @@ -53,18 +53,18 @@ - + - + - + @@ -78,7 +78,7 @@ - preprocessing/python_scripts/ + preprocessing/python_scripts/ renpass_gis/simple_feedin/simple_feedin.py @@ -387,7 +387,7 @@ model_draft.ego_supply_renewable_bnetza_full_attribute - ego_dp_preprocessing_conv_powerplant.sql + ego_dp_preprocessing_conv_powerplant.sql @@ -423,7 +423,7 @@ model_draft.ego_supply_conv_nep2035_temp - ego_dp_preprocessing_res_powerplant.sql + ego_dp_preprocessing_res_powerplant.sql @@ -693,10 +693,10 @@ model_draft.ego_supply_renewable_bnetza_full_attribute + - @@ -710,10 +710,10 @@ model_draft.ego_grid_hv_electrical_neighbours_bus (DP)supply.ego_renewable_powerplant + - @@ -727,10 +727,10 @@ model_draft.ego_grid_hv_electrical_neighbours_bus (DP)supply.ego_conventional_powerplant + - @@ -1110,7 +1110,6 @@ model_draft.ego_grid_hv_electrical_neighbours_bus (DP) - @@ -1121,7 +1120,6 @@ model_draft.ego_grid_hv_electrical_neighbours_bus (DP) - @@ -1132,7 +1130,6 @@ model_draft.ego_grid_hv_electrical_neighbours_bus (DP) - @@ -1143,7 +1140,6 @@ model_draft.ego_grid_hv_electrical_neighbours_bus (DP) - @@ -1154,7 +1150,6 @@ model_draft.ego_grid_hv_electrical_neighbours_bus (DP) - @@ -1165,7 +1160,6 @@ model_draft.ego_grid_hv_electrical_neighbours_bus (DP) - @@ -1176,7 +1170,6 @@ model_draft.ego_grid_hv_electrical_neighbours_bus (DP) - @@ -1187,7 +1180,6 @@ model_draft.ego_grid_hv_electrical_neighbours_bus (DP) - @@ -1198,7 +1190,6 @@ model_draft.ego_grid_hv_electrical_neighbours_bus (DP) - diff --git a/documentation/bpmn/ego_dp-pre_bpmn_section_wind_potential_areas.graphml b/documentation/bpmn/ego_dp-pre_bpmn_section_wind_potential_areas.graphml index 65920424..51540c26 100644 --- a/documentation/bpmn/ego_dp-pre_bpmn_section_wind_potential_areas.graphml +++ b/documentation/bpmn/ego_dp-pre_bpmn_section_wind_potential_areas.graphml @@ -17,11 +17,11 @@ - + - eGo Pre-Processing - wind potential areas - + eGo Pre-Processing - wind potential areas + @@ -31,40 +31,40 @@ + + + - - - - - + + - + - + - + @@ -181,10 +181,10 @@ supply.vernetzen_wind_potential_area + - @@ -214,10 +214,10 @@ vernetzen_wind_potential_areas.zip + - @@ -247,10 +247,10 @@ supply.vernetzen_wind_potential_area + - @@ -328,7 +328,6 @@ - @@ -339,7 +338,6 @@ - @@ -350,7 +348,6 @@ - @@ -361,7 +358,6 @@ - diff --git a/documentation/bpmn/ego_dp-pre_bpmn_section_zensus.graphml b/documentation/bpmn/ego_dp-pre_bpmn_section_zensus.graphml index 3f242694..f9c51b9e 100644 --- a/documentation/bpmn/ego_dp-pre_bpmn_section_zensus.graphml +++ b/documentation/bpmn/ego_dp-pre_bpmn_section_zensus.graphml @@ -17,11 +17,11 @@ - + - eGo Pre-Processing - zensus - + eGo Pre-Processing - zensus + @@ -31,29 +31,29 @@ + + + - - - - - + + - + @@ -64,7 +64,7 @@ - + @@ -301,10 +301,10 @@ Statistisches Bundesamt, Wiesbaden, Genesis-Online, 2016; Datenlizenz by-2-0tmp_destatis.zensus_2011_pop_per_ha + - @@ -334,10 +334,10 @@ Statistisches Bundesamt, Wiesbaden, Genesis-Online, 2016; Datenlizenz by-2-0Zensus_Bevoelkerung_100m-Gitter.zip + - @@ -369,10 +369,10 @@ Statistisches Bundesamt, Wiesbaden, Genesis-Online, 2016; Datenlizenz by-2-0 + - @@ -496,7 +496,6 @@ Statistisches Bundesamt, Wiesbaden, Genesis-Online, 2016; Datenlizenz by-2-0 - @@ -507,7 +506,6 @@ Statistisches Bundesamt, Wiesbaden, Genesis-Online, 2016; Datenlizenz by-2-0 - @@ -518,7 +516,6 @@ Statistisches Bundesamt, Wiesbaden, Genesis-Online, 2016; Datenlizenz by-2-0 - @@ -529,7 +526,6 @@ Statistisches Bundesamt, Wiesbaden, Genesis-Online, 2016; Datenlizenz by-2-0 - @@ -540,7 +536,6 @@ Statistisches Bundesamt, Wiesbaden, Genesis-Online, 2016; Datenlizenz by-2-0 - @@ -551,7 +546,6 @@ Statistisches Bundesamt, Wiesbaden, Genesis-Online, 2016; Datenlizenz by-2-0 - diff --git a/documentation/bpmn/ego_dp-pre_bpmn_sections.graphml b/documentation/bpmn/ego_dp-pre_bpmn_sections.graphml index 69f59134..a07044ad 100644 --- a/documentation/bpmn/ego_dp-pre_bpmn_sections.graphml +++ b/documentation/bpmn/ego_dp-pre_bpmn_sections.graphml @@ -17,11 +17,11 @@ - + - eGo Data-PreProcessing - + eGo Data-PreProcessing + @@ -53,18 +53,18 @@ - + - + - + diff --git a/documentation/bpmn/ego_dp_bpmn_section_postprocessing.graphml b/documentation/bpmn/ego_dp_bpmn_section_postprocessing.graphml index 4abb7506..88407592 100644 --- a/documentation/bpmn/ego_dp_bpmn_section_postprocessing.graphml +++ b/documentation/bpmn/ego_dp_bpmn_section_postprocessing.graphml @@ -17,11 +17,11 @@ - + eGo DataProcessing - postprocessing - + @@ -53,13 +53,13 @@ - + - + @@ -243,10 +243,10 @@ model_draft.ego_grid_line_expansion_costs BBPlG + - @@ -260,10 +260,10 @@ model_draft.ego_grid_line_expansion_costs EnLAG + - @@ -277,10 +277,10 @@ model_draft.ego_grid_line_expansion_costs OPSD + - @@ -294,10 +294,10 @@ model_draft.ego_grid_line_expansion_costs renpassGIS + - @@ -311,10 +311,10 @@ model_draft.ego_grid_line_expansion_costs osmTGmod + - @@ -329,10 +329,10 @@ model_draft.ego_grid_line_expansion_costs NEP/dena + - @@ -400,7 +400,6 @@ NEP/dena - @@ -411,7 +410,6 @@ NEP/dena - @@ -422,7 +420,6 @@ NEP/dena - @@ -433,7 +430,6 @@ NEP/dena - @@ -444,7 +440,6 @@ NEP/dena - @@ -455,7 +450,6 @@ NEP/dena - @@ -466,7 +460,6 @@ NEP/dena - diff --git a/documentation/bpmn/ego_dp_bpmn_section_versioning.graphml b/documentation/bpmn/ego_dp_bpmn_section_versioning.graphml index 931fe27b..0ba14e9f 100644 --- a/documentation/bpmn/ego_dp_bpmn_section_versioning.graphml +++ b/documentation/bpmn/ego_dp_bpmn_section_versioning.graphml @@ -17,11 +17,11 @@ - + - eGo DataProcessing - versioning - + eGo DataProcessing - versioning + @@ -53,18 +53,18 @@ - + - + - + diff --git a/documentation/doc/_static/custom.css b/documentation/doc/_static/custom.css new file mode 100644 index 00000000..5bdc6a00 --- /dev/null +++ b/documentation/doc/_static/custom.css @@ -0,0 +1,3 @@ +.wy-nav-content { + max-width: 100% !important; +} \ No newline at end of file diff --git a/documentation/doc/conf.py b/documentation/doc/conf.py index 6678a675..719dc7ae 100644 --- a/documentation/doc/conf.py +++ b/documentation/doc/conf.py @@ -48,16 +48,17 @@ 'sphinx.ext.intersphinx', 'sphinx.ext.todo', 'sphinx.ext.coverage', - 'sphinx.ext.imgmath' , + 'sphinx.ext.imgmath', 'sphinx.ext.viewcode', -# 'sphinx.ext.autosummary', -# 'sphinxcontrib.napoleon',#enable Napoleon interpreter of docstrings Sphinx v<=1.2 - 'sphinx.ext.napoleon', #enable Napoleon Sphinx v>1.3 -# 'sphinx_paramlinks',#to have links to the types of the parameters of the functions - 'numpydoc', - 'sphinxcontrib.httpdomain', # for restfull API - 'sphinxcontrib.autohttp.flask', - 'sphinx.ext.extlinks' # enables external links with a key + # 'sphinx.ext.autosummary', + # 'sphinxcontrib.napoleon',#enable Napoleon interpreter of docstrings Sphinx v<=1.2 + 'sphinx.ext.napoleon', # enable Napoleon Sphinx v>1.3 + # 'sphinx_paramlinks',#to have links to the types of the parameters of the functions + 'numpydoc', + 'sphinxcontrib.httpdomain', # for restfull API + 'sphinxcontrib.autohttp.flask', + 'sphinx.ext.extlinks', # enables external links with a key + 'sphinx.ext.autosectionlabel' ] # Napoleon settings @@ -235,7 +236,7 @@ def __getattr__(cls, name): # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". -# html_static_path = ['_static'] +html_static_path = ['_static'] # Add any extra paths that contain custom files (such as robots.txt or # .htaccess) here, relative to this directory. These files are copied @@ -300,6 +301,11 @@ def __getattr__(cls, name): # Output file base name for HTML help builder. htmlhelp_basename = 'eGodoc' +# Customize layout with CSS +html_css_files = [ + 'custom.css', +] + # -- Options for LaTeX output --------------------------------------------- latex_elements = { diff --git a/documentation/doc/getting_started.rst b/documentation/doc/getting_started.rst index dac9389a..95ca080d 100644 --- a/documentation/doc/getting_started.rst +++ b/documentation/doc/getting_started.rst @@ -1,43 +1,95 @@ +Getting started =============== -Getting Started -=============== - -Overview -******** -- How to install -- How to run the dataprocessing package -Installation -============ +Installation and setup +---------------------- Installation of latest release -``` -(sudo) pip3 install dataprocessing -``` +.. code-block:: python + + pip install dataprocessing + +or download latest source code + +.. code-block:: python + + git clone git@github.com:openego/data_processing.git + +and install the developer version + +.. code-block:: python -Installation (of developer version) via pip on linux systems as follows + pip install -e -``` -sudo pip3 install -e -``` Troubleshooting +^^^^^^^^^^^^^^^ On Windows -*********** +"""""""""" + +1. **Problem**: Installation of required package shapely fails. **Solution**: Install pre-build +wheel from `here `_. + +Database with docker (optional) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +You need to have `docker-compose` installed. And some more tools, see `requirements_system.txt`. + +Spin up database with docker +"""""""""""""""""""""""""""" + +.. code-block:: bash + + docker-compose up -d --build + +Database credentials +"""""""""""""""""""" + +======== ====== +database dp +port 54321 +user oeuser +password egon +======== ====== + + + +Execute pre-processing +----------------------- + +Execute + +.. code-block:: bash + + python preprocessing/eGo_import_and_preprocessing.py + +for starting pre-processing from scratch doing data import and a pre-processing of imported tables. + +Once completed, export required schemas to SQL dump + +*You might have to install `postgresql-client-11` from additional source to have a compatible client with the v11 +dockered PostgreSQL database.* + +.. code-block:: bash + + /usr/lib/postgresql/11/bin/pg_dump -d dp -U oeuser -p 54321 -h localhost -Fc -n model_draft -n openstreetmap -n boundaries -n society > `date -I`_eGo_data_pre-processing_bremen.backup + +Re-import dumped data by + +.. code-block:: bash + + /usr/lib/postgresql/11/bin/pg_restore -d dp -U oeuser -p 54321 -h localhost 2020-03-28_eGo_data_pre-processing_deu.backup -1. __Problem__: Installation of required package shapely fails. __Solution__: Install pre-build - wheel from [here](http://www.lfd.uci.edu/~gohlke/pythonlibs/#shapely) +You can run data processing by calling command-line script +Execute data processing +------------------------ -Run -==== - You can run data processing by calling command-line script +.. code-block:: bash - ``` - ~$ ego_data_processing - ``` + python dataprocessing/eGo_data_processing.py diff --git a/documentation/doc/images/ego_dp-pre_bpmn_section_bkg-vg250.png b/documentation/doc/images/ego_dp-pre_bpmn_section_bkg-vg250.png new file mode 100644 index 00000000..43e7520b Binary files /dev/null and b/documentation/doc/images/ego_dp-pre_bpmn_section_bkg-vg250.png differ diff --git a/documentation/doc/images/ego_dp-pre_bpmn_section_cosmoclmgrid.png b/documentation/doc/images/ego_dp-pre_bpmn_section_cosmoclmgrid.png new file mode 100644 index 00000000..efc576f3 Binary files /dev/null and b/documentation/doc/images/ego_dp-pre_bpmn_section_cosmoclmgrid.png differ diff --git a/documentation/doc/images/ego_dp-pre_bpmn_section_demand_federalstate.png b/documentation/doc/images/ego_dp-pre_bpmn_section_demand_federalstate.png new file mode 100644 index 00000000..85d6e2d3 Binary files /dev/null and b/documentation/doc/images/ego_dp-pre_bpmn_section_demand_federalstate.png differ diff --git a/documentation/doc/images/ego_dp-pre_bpmn_section_openstreetmap.png b/documentation/doc/images/ego_dp-pre_bpmn_section_openstreetmap.png new file mode 100644 index 00000000..285dd119 Binary files /dev/null and b/documentation/doc/images/ego_dp-pre_bpmn_section_openstreetmap.png differ diff --git a/documentation/doc/images/ego_dp-pre_bpmn_section_osmtgmod.png b/documentation/doc/images/ego_dp-pre_bpmn_section_osmtgmod.png new file mode 100644 index 00000000..2ab8fdfa Binary files /dev/null and b/documentation/doc/images/ego_dp-pre_bpmn_section_osmtgmod.png differ diff --git a/documentation/doc/images/ego_dp-pre_bpmn_section_renpassGIS.png b/documentation/doc/images/ego_dp-pre_bpmn_section_renpassGIS.png new file mode 100644 index 00000000..e3d84c9c Binary files /dev/null and b/documentation/doc/images/ego_dp-pre_bpmn_section_renpassGIS.png differ diff --git a/documentation/doc/images/ego_dp-pre_bpmn_section_scenario-log.png b/documentation/doc/images/ego_dp-pre_bpmn_section_scenario-log.png new file mode 100644 index 00000000..3170a466 Binary files /dev/null and b/documentation/doc/images/ego_dp-pre_bpmn_section_scenario-log.png differ diff --git a/documentation/doc/images/ego_dp-pre_bpmn_section_slp.png b/documentation/doc/images/ego_dp-pre_bpmn_section_slp.png new file mode 100644 index 00000000..916bb1ba Binary files /dev/null and b/documentation/doc/images/ego_dp-pre_bpmn_section_slp.png differ diff --git a/documentation/doc/images/ego_dp-pre_bpmn_section_slp_parameters.png b/documentation/doc/images/ego_dp-pre_bpmn_section_slp_parameters.png new file mode 100644 index 00000000..4aa56207 Binary files /dev/null and b/documentation/doc/images/ego_dp-pre_bpmn_section_slp_parameters.png differ diff --git a/documentation/doc/images/ego_dp-pre_bpmn_section_supply.png b/documentation/doc/images/ego_dp-pre_bpmn_section_supply.png new file mode 100644 index 00000000..ae29b823 Binary files /dev/null and b/documentation/doc/images/ego_dp-pre_bpmn_section_supply.png differ diff --git a/documentation/doc/images/ego_dp-pre_bpmn_section_wind_potential_areas.png b/documentation/doc/images/ego_dp-pre_bpmn_section_wind_potential_areas.png new file mode 100644 index 00000000..87e6ae29 Binary files /dev/null and b/documentation/doc/images/ego_dp-pre_bpmn_section_wind_potential_areas.png differ diff --git a/documentation/doc/images/ego_dp-pre_bpmn_section_zensus.png b/documentation/doc/images/ego_dp-pre_bpmn_section_zensus.png new file mode 100644 index 00000000..1419b1f6 Binary files /dev/null and b/documentation/doc/images/ego_dp-pre_bpmn_section_zensus.png differ diff --git a/documentation/doc/images/ego_dp-pre_bpmn_sections.png b/documentation/doc/images/ego_dp-pre_bpmn_sections.png new file mode 100644 index 00000000..5a025a51 Binary files /dev/null and b/documentation/doc/images/ego_dp-pre_bpmn_sections.png differ diff --git a/documentation/doc/images/ego_dp_bpmn_section_REA.png b/documentation/doc/images/ego_dp_bpmn_section_REA.png new file mode 100644 index 00000000..022e22dd Binary files /dev/null and b/documentation/doc/images/ego_dp_bpmn_section_REA.png differ diff --git a/documentation/doc/images/ego_dp_bpmn_section_griddistrict.png b/documentation/doc/images/ego_dp_bpmn_section_griddistrict.png new file mode 100644 index 00000000..172cfd07 Binary files /dev/null and b/documentation/doc/images/ego_dp_bpmn_section_griddistrict.png differ diff --git a/documentation/doc/images/ego_dp_bpmn_section_input_data_check.png b/documentation/doc/images/ego_dp_bpmn_section_input_data_check.png new file mode 100644 index 00000000..c4f06ec1 Binary files /dev/null and b/documentation/doc/images/ego_dp_bpmn_section_input_data_check.png differ diff --git a/documentation/doc/images/ego_dp_bpmn_section_loadarea.png b/documentation/doc/images/ego_dp_bpmn_section_loadarea.png new file mode 100644 index 00000000..3ad33760 Binary files /dev/null and b/documentation/doc/images/ego_dp_bpmn_section_loadarea.png differ diff --git a/documentation/doc/images/ego_dp_bpmn_section_lowvoltage.png b/documentation/doc/images/ego_dp_bpmn_section_lowvoltage.png new file mode 100644 index 00000000..7d0b897c Binary files /dev/null and b/documentation/doc/images/ego_dp_bpmn_section_lowvoltage.png differ diff --git a/documentation/doc/images/ego_dp_bpmn_section_postprocessing.png b/documentation/doc/images/ego_dp_bpmn_section_postprocessing.png new file mode 100644 index 00000000..6390a41c Binary files /dev/null and b/documentation/doc/images/ego_dp_bpmn_section_postprocessing.png differ diff --git a/documentation/doc/images/ego_dp_bpmn_section_powerflow.png b/documentation/doc/images/ego_dp_bpmn_section_powerflow.png new file mode 100644 index 00000000..56589a60 Binary files /dev/null and b/documentation/doc/images/ego_dp_bpmn_section_powerflow.png differ diff --git a/documentation/doc/images/ego_dp_bpmn_section_substation.png b/documentation/doc/images/ego_dp_bpmn_section_substation.png new file mode 100644 index 00000000..336a0cd1 Binary files /dev/null and b/documentation/doc/images/ego_dp_bpmn_section_substation.png differ diff --git a/documentation/doc/images/ego_dp_bpmn_section_versioning.png b/documentation/doc/images/ego_dp_bpmn_section_versioning.png new file mode 100644 index 00000000..69df7769 Binary files /dev/null and b/documentation/doc/images/ego_dp_bpmn_section_versioning.png differ diff --git a/documentation/doc/images/ego_dp_bpmn_sections.png b/documentation/doc/images/ego_dp_bpmn_sections.png new file mode 100644 index 00000000..dcd03a81 Binary files /dev/null and b/documentation/doc/images/ego_dp_bpmn_sections.png differ diff --git a/documentation/doc/index.rst b/documentation/doc/index.rst index 07a13ab9..234e520b 100644 --- a/documentation/doc/index.rst +++ b/documentation/doc/index.rst @@ -7,9 +7,18 @@ Welcome to Data processing documentation of open_eGo :scale: 100% -[Add text here] +This package contains the scripts of the data processing for the research project +`open_eGo `_. +The data processing creates data required for grid model generators (`osmTGmod `_ +and `ding0 `_) and grid planning calculations conducted with +`eTraGo `_ and `eDisGo `_. +More information about the project can be found in :ref:`Open_eGo the project` and on the +`project website `_. +See :ref:`Getting started` for instructions on installation and execution. +An overview on data processing steps is given in :ref:`Data processing scripts`. + .. toctree:: :maxdepth: 7 @@ -39,14 +48,3 @@ Copyright (C) 2015-2018 open_eGo-Team This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/. - - - - - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` diff --git a/documentation/doc/scripts.rst b/documentation/doc/scripts.rst index 69edfd62..3d0d3eec 100644 --- a/documentation/doc/scripts.rst +++ b/documentation/doc/scripts.rst @@ -2,39 +2,195 @@ Data processing scripts ======================= +The preparation of data in organized into :ref:`Preprocessing` and :ref:`Data processing`. +Scripts contained in the :ref:`Preprocessing` include import of raw and basic preparation for later use in scripts of +:ref:`Data processing`. + +Preprocessing +============= + +Overview on pre-processing scripts that import data and do basic preparation. +Data pre-processing steps partially depend on each other which is shown in the following figure. + +.. image:: images/ego_dp-pre_bpmn_sections.png + :width: 80% + +Scenario log +------------ + +Prepare logging of data processing: create logging table and respective function that logs changes made to tables. + +.. image:: images/ego_dp-pre_bpmn_section_scenario-log.png + :width: 80% + +RenpassG!S +---------- + +Using repassG!S for creating time series of renewable power plants feedin. + +.. image:: images/ego_dp-pre_bpmn_section_renpassGIS.png + :width: 80% + +COSMO-CLM grid +-------------- + +Import spatial data grid for COSMO-CLM climate data. + +.. image:: images/ego_dp-pre_bpmn_section_cosmoclmgrid.png + :width: 70% + +Zensus +------ + +Import Zensus data. + +.. image:: images/ego_dp-pre_bpmn_section_zensus.png + +Import data for administrative boundaries from VG250 data set. + +.. image:: images/ego_dp-pre_bpmn_section_bkg-vg250.png + +OpenStreetMap +------------- + +Import OpenStreetMap (OSM) data from Geofabrik data dumps. + +.. image:: images/ego_dp-pre_bpmn_section_openstreetmap.png + +Standard load profile +--------------------- + +Insert standard load profile (SLP) parameters to database table. + +.. image:: images/ego_dp-pre_bpmn_section_slp.png + :width: 60% + +Wind potential areas +-------------------- + +Import wind potential area data from VerNETZen project. + +.. image:: images/ego_dp-pre_bpmn_section_wind_potential_areas.png + :width: 90% + + +Demand at federal states +------------------------ + +Insert annual demand data at federal state level for different energy use sectors and disaggregate further. + +.. image:: images/ego_dp-pre_bpmn_section_demand_federalstate.png + :width: 80% + +osmTGmod +-------- + +EHV and HV level grid obtained from OSM with osmTGmod. + +.. image:: images/ego_dp-pre_bpmn_section_osmtgmod.png + :width: 60% + +Power plant data +---------------- + +Basic preparation of power plant data. + +.. image:: images/ego_dp-pre_bpmn_section_supply.png + Data processing =============== -SQL-Scripts ------------ +The data processing is organized into separate parts which are executed in desired order. -.. toctree:: - :maxdepth: 1 - :titlesonly: +.. image:: images/ego_dp_bpmn_sections.png - Dataprocessing +Each of these parts consists of multiple data processing steps that are stored in separate SQL files. +The following sections explain what is done in each section, which data is accessed, created or manipulated, and +which tables results of each script. -Python-Scripts --------------- +Input data chech +---------------- -[use sphinx doc strings] +Check of data prepared for HV/EHV power grid calculations. +.. image:: images/ego_dp_bpmn_section_input_data_check.png + :width: 800px -Preprocessing -============= +Substation +---------- + +Process data for sub-stations in grid levels from EHV to MV and create Voronoi cuts around HV sub-stations. + +.. image:: images/ego_dp_bpmn_section_substation.png -SQL-Scripts +Grid district +------------- + +Derive medium-voltage grid districts from administrative boundaries and HV/MV sub-station locations. + +.. image:: images/ego_dp_bpmn_section_griddistrict.png + + +Load area +--------- + +Define areas of electricity consumption categorized into energy consumption sectors. + +.. image:: images/ego_dp_bpmn_section_loadarea.png + +Low-voltage ----------- + +Estimate medium/low-voltage sub-station locations and respective low-voltage grid districts. + +.. image:: images/ego_dp_bpmn_section_lowvoltage.png + +Renewable energy allocation (REA) +--------------------------------- + +Enrich renewable power plant data base with more plausible locations for power plants with unprecise information about +the location. + +.. image:: images/ego_dp_bpmn_section_REA.png + + +Power flow +---------- + +Prepare PyPSA-compatible tables for power-flow calculation in the EHV/HV grid level. + +.. image:: images/ego_dp_bpmn_section_powerflow.png + +Post-processing +--------------- + +Post-process :ref:`Power flow` data. + + +.. image:: images/ego_dp_bpmn_section_postprocessing.png + :width: 800px + +Versioning +---------- + +Move resulting data to versioned schemas of the database. + +.. image:: images/ego_dp_bpmn_section_versioning.png + + + + +Additional information on SQL-Scripts +===================================== + .. toctree:: :maxdepth: 1 :titlesonly: + Dataprocessing Preprocessing -Python-Scripts --------------- -[use sphinx doc strings] diff --git a/documentation/requirements.txt b/documentation/requirements.txt index b7a36882..c005edbe 100644 --- a/documentation/requirements.txt +++ b/documentation/requirements.txt @@ -2,7 +2,7 @@ # Using single requirments for docs, see: # https://github.com/rtfd/readthedocs.org/issues/2070 sphinx_rtd_theme -numpy == 1.12.1 -numpydoc == 0.7.0 +numpy +numpydoc sphinxcontrib-httpdomain aiohttp_jinja2