Create python module to validate processed data #18

horstf · 2022-04-13T08:21:40Z

No description provided.

…rization nameing schema

joergfunger

Not sure if the merge request is uptodate, at least test files are currently not working.

joergfunger · 2022-07-18T15:26:41Z

@horstf we have updated the structure, could you try to adapt your code the way it is now working in this new structure, such that we could merge it into main?

horstf · 2022-07-19T09:51:32Z

@horstf we have updated the structure, could you try to adapt your code the way it is now working in this new structure, such that we could merge it into main?

Is that the structure in the /lebedigital folder? I.e. refactor my code in an appropriate /lebedigital/validation folder?

horstf · 2022-07-26T09:37:32Z

I've copied the validation submodule into the new lebedigital package. I haven't updated the dodo.py file yet because the current version is not using the new package for now.

joergfunger

thanks for the pull request, there are some modifications required to adapt to the new structure and some documentation issues.

lebedigital/validation.py

joergfunger · 2022-08-03T11:53:59Z

usecases/Concrete/knowledgeGraph/emodul/validation.py

@@ -0,0 +1,78 @@
+from pyshacl import validate


not sure that that file does, but all files should eiter be in lebedigital (the actual functions), in tests (the tests for these functions) or in minimumWorkingExample (the pydoit workflow that processes the data for the minimum working example). The folder usecases/Concrete is deprecated.

That file is not used anymore, I think that was an accidental commit

This file is still in Concrete, please remove it?

joergfunger

Thanks for the changes, I still have a few comments. Where are the tests related to validation.py? In addition, the folder usecases/Concrete is deprecated.

joergfunger · 2022-08-31T08:49:18Z

usecases/Concrete/knowledgeGraph/emodul/validation.py

@@ -0,0 +1,78 @@
+from pyshacl import validate


This file is still in Concrete, please remove it?

joergfunger · 2022-08-31T08:50:33Z

usecases/Concrete/knowledgeGraph/requirements.txt

@@ -8,5 +8,6 @@ SPARQLWrapper==1.8.5
 requests==2.22.0


This file is also outdated, the conda environment at the top level is the one and only to be used.

We can ignore these commits then, all the libraries I need are already part of the environment file.

I've also deleted the validation.py from Concrete

@joergfunger For the tests, is there a graph that I can use to test my code? The usecases/MinimumWorkingExample/emodul/processed_data folder is empty but I would need an EM_Graph.ttl file or so to test the validation. I can also just create my own otherwise from the data.

Could you talk to @PoNeYvIf on that? For the tests, it would also be possible (and maybe even better) to create your own very simple test cases (it does not have to be related to a very specific ontology, but rather a very general one such as foaf or prov). In particular make sure that there are errors existing such that those can be returned and processed. We could also have a short phone call on that.

E.g. if we know that we test for machine ids via shacl, we could get the machine IDs from the global KG that are referred to in the local KG. This would of course mean that every time we add a rule we have to think about the SPARQL query and the information that could be contained globally and not locally, i don't know if this is feasible.

Yes, let's try to create the queries, let me know when it's okay for you.

Not sure if I would always download the complete global graph. It would rather be necessary if particular instances that are referred to in the local graph (e.g. the testing machine that has already be created, or the mix_design we are referring to when adding a new Youngs modulus test) is existing. So for the specific tests, we would first generate the mix design, and upload it to the global KG. Then we would create a Youngs modulus test that references an instance of the mix design (without creating the same instance again). By uploading that then to the KG, these two data sets (mix design and Youngs modulus test) are automatically connected. And before uploading the Youngs modulus KG, we would have to check, if this instance of the mix design is already existing in the global KG (and potentially further properties apart from the id, but that is not necessary right now).

As for the sparql query, we somehow know in the local KG generation what classes we expect. So the sparql query should return the ids of all instances of a given class - that should be quite general and would not require rewriting the sparql query each time we have a new test.

Maybe we should have a talk together with @firmao to discuss the specifics?

…tion

firmao · 2022-10-14T09:06:41Z

Sure, please, send a message on teams.

…

On Fri, Oct 14, 2022 at 10:30 AM horstf ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In usecases/Concrete/knowledgeGraph/requirements.txt <#18 (comment)> : > @@ -8,5 +8,6 @@ SPARQLWrapper==1.8.5 requests==2.22.0 Maybe we should have a talk together with @firmao <https://github.com/firmao> to discuss the specifics? — Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACGR4RMUMLQAUC4PSISZ5ZLWDEK2FANCNFSM5TJ3K5IA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

joergfunger · 2022-11-17T15:17:11Z

@horstf What is the status with this merge request? It seems that only the integration into the dodo file in minimum working example is missing.

joergfunger · 2022-12-18T06:31:27Z

@horstf The actions are not working, so that prevents me from merging the branch. In addition, I checked the other files. I guess there was a misunderstanding with the dodo.py, since you changed that in concrete. I guess we would have to remove this directory (it was already deprecated, but not removed, because it included all the old stuff). The actual dodo.py we are working with is in usecases/MinimumWorkingExample. Could you update that there as well. I guess the problem is that we do not have a working generation of the KG there yet, @AidaZt @soudehMasoudian @alFrie how are we going to handle this merge request? Or should @horstf integrate that in pydoit without actually including in the tasks, such that you could add that afterwards once the KG is up and running?

horstf · 2023-01-10T11:44:12Z

Hi, is there any update on how i should proceed?

joergfunger · 2023-01-10T12:07:48Z

Could we have a short teams meeting? Maybe that is easier

horstf · 2023-01-10T14:41:17Z

Could we have a short teams meeting? Maybe that is easier

Yes, the date you suggested in your teams invitation works fine.

horstf added 2 commits April 13, 2022 10:17

add modularized validation

8c1dd85

add shape to test

d181201

horstf changed the title ~~Validation modularization~~ Create python module to validate processed data Apr 13, 2022

horstf added 5 commits April 13, 2022 10:24

add old dodo.py code from ont_dev branch (have to update)

fc37a5e

update requirements and environment

68aed79

add __init__.py for modularization of code

f49e040

rename emodul_validation.py to validation.py to imporve python modula…

644a4c9

…rization nameing schema

add validation to dodo.py

f752d17

horstf marked this pull request as ready for review May 3, 2022 08:41

horstf requested a review from joergfunger May 3, 2022 08:41

horstf added 2 commits May 3, 2022 13:08

add shape as dep to dodo.py

c4dd67b

move and rename shape

230ac90

joergfunger reviewed May 5, 2022

View reviewed changes

horstf marked this pull request as draft May 6, 2022 10:01

horstf added 4 commits July 26, 2022 11:17

merge main

458da5e

move validation into own lebedigital submodule

1e3697e

validation didnt need additional submodule structure

f35255b

add comments

379c107

horstf marked this pull request as ready for review July 26, 2022 10:04

horstf requested a review from joergfunger August 3, 2022 10:11

joergfunger requested changes Aug 3, 2022

View reviewed changes

horstf added 2 commits August 4, 2022 13:42

add comments

2a3e61c

remove superflous comments

b9b3175

horstf requested a review from joergfunger August 24, 2022 11:48

joergfunger requested changes Aug 31, 2022

View reviewed changes

remove unused file from concrete

c150d15

joergfunger mentioned this pull request Sep 6, 2022

Join multiple knowledgegraphs #48

Open

horstf added 11 commits September 7, 2022 16:34

add files to test validation

8b25331

add validation tests

da2cebf

add validation tests

de56554

remove empty lines

1a49eab

add validation tests

26e0925

undo changes in requirements.txt

9e92d70

undo changes in requirements.txt

3133735

Merge remote-tracking branch 'origin/main' into validation_modulariza…

f0d889a

…tion

add some dodo code

0d501bc

add output code

3cd8a72

add output code

a3917f9

joergfunger linked an issue Dec 5, 2022 that may be closed by this pull request

shacl validation #93

Open

update dodo

2678023

horstf closed this Jan 16, 2023

horstf deleted the validation_modularization branch January 16, 2023 13:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create python module to validate processed data #18

Create python module to validate processed data #18

horstf commented Apr 13, 2022

joergfunger left a comment

joergfunger commented Jul 18, 2022

horstf commented Jul 19, 2022

horstf commented Jul 26, 2022

joergfunger left a comment

joergfunger Aug 3, 2022

horstf Aug 4, 2022

joergfunger Aug 31, 2022

joergfunger left a comment

joergfunger Aug 31, 2022

joergfunger Aug 31, 2022

horstf Sep 5, 2022

horstf Sep 5, 2022

horstf Sep 5, 2022 •

edited

Loading

joergfunger Sep 5, 2022

horstf Oct 11, 2022

firmao Oct 11, 2022

joergfunger Oct 11, 2022

joergfunger Oct 11, 2022 •

edited

Loading

horstf Oct 14, 2022

firmao commented Oct 14, 2022 via email

joergfunger commented Nov 17, 2022

joergfunger commented Dec 18, 2022

horstf commented Jan 10, 2023

joergfunger commented Jan 10, 2023

horstf commented Jan 10, 2023

Create python module to validate processed data #18

Create python module to validate processed data #18

Conversation

horstf commented Apr 13, 2022

joergfunger left a comment

Choose a reason for hiding this comment

joergfunger commented Jul 18, 2022

horstf commented Jul 19, 2022

horstf commented Jul 26, 2022

joergfunger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joergfunger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

horstf Sep 5, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joergfunger Oct 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

firmao commented Oct 14, 2022 via email

joergfunger commented Nov 17, 2022

joergfunger commented Dec 18, 2022

horstf commented Jan 10, 2023

joergfunger commented Jan 10, 2023

horstf commented Jan 10, 2023

horstf Sep 5, 2022 •

edited

Loading

joergfunger Oct 11, 2022 •

edited

Loading