Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Review: open_MODEX-base_scenario #71

Open
10 of 15 tasks
jh-RLI opened this issue Oct 8, 2020 · 8 comments · Fixed by #87 · May be fixed by #73
Open
10 of 15 tasks

Review: open_MODEX-base_scenario #71

jh-RLI opened this issue Oct 8, 2020 · 8 comments · Fixed by #87 · May be fixed by #73
Assignees
@Ludee @jh-RLI @christian-rli
Labels
review Data and Metadata review v1.5.1 metadata created with version 1.5.1

Comments

@jh-RLI
Copy link
Contributor

jh-RLI commented Oct 8, 2020 •
edited by steull
Loading

Issue description

I'm submitting an open dataset with a corresponding OEMetadata string for review.

The oedatamodel is a collection of multiple tables that are described in the resources. open_MODEX uses the oedatamodel to store input and output related data in the oedatamodel tables. The tables are used as a shared homogonous data source for the framework comparison.

Workflow checklist

  1. GitHub
  • I have submitted this issue to have metadata and data review documented (Issue Review: open_MODEX-base_scenario #71 )
  • Create a new review-branch and push OEMetadata to new branch (review/open_MODEX-base_scenario#71)
  1. OEP
  • Upload data to the OEP in schema model_draft (see upload tutorial)
  • Link URL of data in this issue (model_draft.project_nameofdata)
  1. Start a Review
  • Start a pull request (PR) from review-branch to master
  • Assign a reviewer and get in contact
  1. Reviewer section
  • A reviewer starts working on the issue
  • Review data license
  • A reviewer finished working on this issue (and awarded a badge)
  • Update metadata on table
  • Data moved to its final schema
  • Add OEP tags to table
  • Merge PR and delete review-branch
  • Document final links of metadata and data in issue description
  • Close issue

Metadata and data for review

Here are the links to my data and metadata. Naming follows the pattern model_draft.project_nameofdata:
Metadata:

Data: https://openenergy-platform.org/dataedit/view/model_draft/project_nameofdata

Reviewed and published metadata and data

Final naming and location of the data and metadata after the review are as follows:
schema.tablename
@jh-RLI jh-RLI added the review Data and Metadata review label Oct 8, 2020
@jh-RLI jh-RLI linked a pull request Oct 14, 2020 that will close this issue
Open
@jh-RLI jh-RLI linked a pull request Oct 14, 2020 that will close this issue
review/open_MODEX base scenario datapackage #73
Open
christian-rli added a commit that referenced this issue Oct 15, 2020
@christian-rli
Replace empty strings with null #71
8f998c9
christian-rli added a commit that referenced this issue Oct 15, 2020
@christian-rli
Fix empty foreignKey section #71
2d17801
christian-rli added a commit that referenced this issue Oct 15, 2020
@christian-rli
Fix typo #71
c6aef36
@christian-rli
Copy link
Contributor

metadata can now be parsed by omi. content looks good. (When) will there be any data uploaded to the oep?

@jh-RLI
Copy link
Contributor Author

jh-RLI commented Oct 15, 2020

The upload is in progress. ATM we only have some incomplete data sets on the OEP. There are still some minor data-related issues.

@Ludee Ludee linked a pull request Nov 30, 2021 that will close this issue
Metadata for MODEX consortium #87
Merged
@christian-rli
Copy link
Contributor

I noticed that the json covers several resources. However not all 4 tables that can be found when searching modex on the OEP are covered. Specifically oed_data is missing. At the same time, the three resources are listed in the metadata of each individual table, which I think is technically wrong. The tables might belong together, but the table description should reference only values of the table it describes. Contentwise everything looks ok, as far as i can tell.

@christian-rli
Copy link
Contributor

It's more complicated than I first thought, actually. There's a separate table for values, it appears. It is named oed_scalar and has resource descriptions in the metadata. However the table oed_data matches the resource description under oed_concrete_scalar. This is a bit messed up. I could see the point of having several resource descriptions under one table, if all tables belong together, but this simply isn't accurate.

@christian-rli
Copy link
Contributor

I cannot just download and reupload a new table, because i"m missing the column descriptions to create the new table (due to the same faulty metadata). I also have no time to write new column descriptions for the upload until tomorrow, because my train just arrived. I recommend showing the tables as is in the current directory. They have metadata that is largely correct. Fixing the structure is beyond what I can deliver today. Let me know if I can still help tomorrow forenoon @Ludee . Maybe @jh-RLI can either do a quick fix, or explain why this makes sense the way it is.

@Ludee
Copy link
Member

Ludee commented Nov 30, 2021

I guess that's the problem with the OEDatamodel. It has two formats and the metadata has to fit both.
Thanks for your help!

@jh-RLI
Copy link
Contributor Author

jh-RLI commented Nov 30, 2021

That's right, we used the oed-concrete format to create the metadata because that's the format a user should apply. The missing table only appears in the oed-normalization format. The normalization format is meant to be used on the database.
After creating the metadata, I didn't think to adjust the resources. I will adjust the resources to the normalization format. @Ludee @christian-rli

@jh-RLI
Copy link
Contributor Author

jh-RLI commented Nov 30, 2021

Should be fixed.

Ludee added a commit that referenced this issue Dec 1, 2021
@Ludee
Review and update contex #71
3ffd151
Ludee added a commit that referenced this issue Dec 1, 2021
@Ludee
Review and update sources #71
c5996e7
@Ludee Ludee closed this as completed in #87 Dec 9, 2021
@jh-RLI jh-RLI reopened this Mar 1, 2022
@steull steull added the v1.4 metadata created with version 1.4 label Mar 1, 2022
Ludee added a commit that referenced this issue Mar 30, 2022
@Ludee
Move new files to correct path #71
d2a287f
@Ludee Ludee mentioned this issue Mar 30, 2022
Merged
steull added a commit that referenced this issue Jul 4, 2022
@steull
Update metadata to v1.5.1 #71
675decc
steull added a commit that referenced this issue Jul 4, 2022
@steull
Removed v1.4 files in old folder #71
9282a98
@steull steull added v1.5.1 metadata created with version 1.5.1 and removed v1.4 metadata created with version 1.4 labels Jul 4, 2022
steull added a commit that referenced this issue Jul 14, 2022
@steull
Changed keywords #71
c294434
steull added a commit that referenced this issue Jul 14, 2022
@steull
changed license names, if necessary and added contributor #71
c352e0d
steull added a commit that referenced this issue Jul 14, 2022
@steull
Changed type in some fields #71
6d4c7a1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Ludee Ludee

@jh-RLI jh-RLI

@christian-rli christian-rli

Labels
review Data and Metadata review v1.5.1 metadata created with version 1.5.1
Projects
None yet
Milestone
No milestone
4 participants
@Ludee @jh-RLI @christian-rli @steull