Review: fh_iee_trafo_fw #75

Richard-Schmitz · 2020-10-29T08:29:33Z

Issue description

I'm submitting an open dataset with a corresponding OEMetadata string for review.
The dataset shows the results from project "Fh IEE Trafo FW" ("Pathways of transformation in the heating sector") of Fraunhofer IEE. It contains a total of 10 different input tables as well as 15 output tables for each of the 4 scenarios.

Workflow checklist

GitHub

I have submitted this issue to have metadata and data review documented (Issue Review: Project "Fh IEE Trafo FW" #74)
Create a new review-branch and push OEMetadata to new branch (review/project_nameofdata#NR)

OEP

Upload data to the OEP in schema model_draft (see upload tutorial)
Link URL of data in this issue (model_draft.project_nameofdata)

Start a Review

Start a pull request (PR) from review-branch to master
Assign a reviewer and get in contact

Reviewer section

A reviewer starts working on the issue
Review data license
A reviewer finished working on this issue (and awarded a badge)
Update metadata on table
Data moved to its final schema
Add OEP tags to table
Merge PR and delete review-branch
Document final links of metadata and data in issue description
Close issue

Metadata and data for review

Here are the links to my data and metadata. Naming follows the pattern model_draft.project_nameofdata:

Metadata:
PR: #74

v1.5.1 https://github.com/OpenEnergyPlatform/data-preprocessing/tree/review/fh_iee_trafo_fw/data-review/75-fh_iee_trafo_fw

Data:

Input data:

Output data:

Reviewed and published metadata and data

Final naming and location of the data and metadata after the review are as follows:
schema.tablename

in each case a datestring conforming to this format is asked: 2020-12-31

christian-rli · 2020-12-02T15:49:48Z

I had to make a few changes so omi could parse the string. Mainly these were replacing "null" with null, "none" with null and deleting empty contributors info. One thing affecting content needed a change: the start and end date of the timeseries weren't datee. They were strings of two dates which both didn't conform to ISO 8601. I changed these to the earliest date for start and the latest date for end. Please let me know if that's a broad mischaracterisation. I'm assuming that the files describe two timeseries in different files? Other than that the metadata are good to go from my perspective. (On a side note: Yay vim for allowing search and replace in a long list of files!)

Richard-Schmitz · 2020-12-03T09:31:30Z

Thanks for the review, your changes work out fine. Concerning the timeseries, it is always one spreadsheet containing data for both 2030 and 2050, so the earliest date for start and the latest date for end are correct now.

christian-rli · 2020-12-08T19:04:51Z

@MGlauer or @meisam-booshehri I'm trying to update the metadata on these corresponding tables, but the OEP tells me that I don't have permission. Neither the webinterface nor the API allow write access. Do you have any idea why that is? Thank you for any information.

Traceback (most recent call last): File "fh-meta.py", line 19, in <module> cl.update_metadata(metadata=data) File "/home/christianh/anaconda3/lib/python3.7/site-packages/oep_client/client.py", line 133, in update_metadata self.request("POST", url, jsondata=metadata) File "/home/christianh/anaconda3/lib/python3.7/site-packages/oep_client/client.py", line 53, in request raise Exception(err) Exception: {'detail': 'You do not have permission to perform this action.'}

MGlauer · 2020-12-08T19:09:16Z

Well, probably because you do not have write permissions. The admin of the table - most probably the creator - has to grant those to you.

christian-rli · 2020-12-08T19:22:51Z

Thank you @MGlauer . It was my impression that tables created via the api are all created by the same user and can therefore be modified(?). It does sound wrong when writing it out though.

Has this changed? I believe, I used to be able to write metadata to other people's tables. Also, how would one grant write access?

... I checked a few of my tables and it looks like I don't have write access to those either. Is my database user connected to my api user? I will need some sort of access in any case for moving the table to its final schema.

@Richard-Schmitz: Could you grant me write access, please? My OEP user name is "Christian Hofmann" And asking out of interest: how were the tables created?

christian-rli · 2020-12-08T19:29:50Z

Never mind the question on how to grant permission rights. The OEP has a big red button for that which I never noticed. Also I found a table that I created and I have access to, so it seems that just database user is not connected to my oep user account and I have created most of my tables with the former.

christian-rli · 2020-12-17T18:49:40Z

@MGlauer I moved the tables to the schema "scenario" and now all tags are lost. This happened before in #27 . I recall that I had to run a command via sql to bring them back, but it wasn't documented anywhere. Could you let me know (again), please? (This time I shall document it properly at an appropriate location)

This shouldn't have to happen. I suppose when moving tables via API this wouldn't happen, but API users aren't allowed to move tables, as far as I am aware. There should be a fix for this in the long term.

steull · 2022-04-07T10:31:21Z

I have updated all metadata from version 1.4 to version 1.5.1. I noticed that the characters for umlauts (for example) are not parsed. I assume these are not ASCII characters? I haven't changed that yet.
I also noticed that all column names are capitalized. Here, the OEP requires all words to be lower case. (I assume we can change that so you don't have to create the tables and upload the data again)
Furthermore, I changed the source to "null" since "own project" is not a valid source. I also made minor adjustments like spatial resolution and timeseries resolution.

Tables moved to scenario schema and all tags are added to the table.
Review process is finished.

Richard-Schmitz added the review Data and Metadata review label Oct 29, 2020

Richard-Schmitz assigned Ludee, jh-RLI and christian-rli Oct 29, 2020

Richard-Schmitz changed the title ~~Review: NameOfDataset~~ Review: fh_iee_trafo_fw Oct 29, 2020

christian-rli added a commit that referenced this issue Dec 2, 2020

Start and End don't always conform to spec #75

111cd74

in each case a datestring conforming to this format is asked: 2020-12-31

christian-rli added a commit that referenced this issue Dec 2, 2020

Replace null and remove foreign key #75

3d99e90

christian-rli added a commit that referenced this issue Dec 2, 2020

Replace all "null" with null #75

399e240

christian-rli added a commit that referenced this issue Dec 2, 2020

Replace all "none" with null #75

b5f3b65

christian-rli added a commit that referenced this issue Dec 2, 2020

Replace all foreignKeys #75

2cc5aed

christian-rli added a commit that referenced this issue Dec 2, 2020

Replace all start and end dates in timeseries #75

73f2402

christian-rli added a commit that referenced this issue Dec 2, 2020

Remove all empty contributors #75

9c9743d

christian-rli mentioned this issue Dec 7, 2020

Not possible to describe interrupted timeseries OpenEnergyPlatform/oemetadata#44

Closed

christian-rli added a commit that referenced this issue Dec 17, 2020

Remove rogue " #75

1358ee0

christian-rli added a commit that referenced this issue Dec 17, 2020

Award badge "gold" #75

1cd4a7c

christian-rli added a commit that referenced this issue Dec 17, 2020

Move to final schema scenario #75

eae5e3f

steull added the v1.4 metadata created with version 1.4 label Mar 1, 2022

steull added a commit that referenced this issue Apr 6, 2022

Moved files with v1.4 in new folder #75

ff2a788

steull added a commit that referenced this issue Apr 6, 2022

delete files v1.4 which are not in new folder #75

b5a513d

steull added a commit that referenced this issue Apr 6, 2022

Added new files with v1.5.1 #75

324ed9c

steull closed this as completed Apr 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Review: fh_iee_trafo_fw #75

Review: fh_iee_trafo_fw #75

Richard-Schmitz commented Oct 29, 2020 •

edited by steull

Loading

christian-rli commented Dec 2, 2020 •

edited

Loading

Richard-Schmitz commented Dec 3, 2020

christian-rli commented Dec 8, 2020 •

edited

Loading

MGlauer commented Dec 8, 2020

christian-rli commented Dec 8, 2020 •

edited

Loading

christian-rli commented Dec 8, 2020 •

edited

Loading

christian-rli commented Dec 17, 2020 •

edited

Loading

steull commented Apr 7, 2022

Review: fh_iee_trafo_fw #75

Review: fh_iee_trafo_fw #75

Comments

Richard-Schmitz commented Oct 29, 2020 • edited by steull Loading

Issue description

Workflow checklist

Metadata and data for review

Reviewed and published metadata and data

christian-rli commented Dec 2, 2020 • edited Loading

Richard-Schmitz commented Dec 3, 2020

christian-rli commented Dec 8, 2020 • edited Loading

MGlauer commented Dec 8, 2020

christian-rli commented Dec 8, 2020 • edited Loading

christian-rli commented Dec 8, 2020 • edited Loading

christian-rli commented Dec 17, 2020 • edited Loading

steull commented Apr 7, 2022

Richard-Schmitz commented Oct 29, 2020 •

edited by steull

Loading

christian-rli commented Dec 2, 2020 •

edited

Loading

christian-rli commented Dec 8, 2020 •

edited

Loading

christian-rli commented Dec 8, 2020 •

edited

Loading

christian-rli commented Dec 8, 2020 •

edited

Loading

christian-rli commented Dec 17, 2020 •

edited

Loading