Df dev with cfgURL and datacatalog in release graph #57

fils · 2024-11-15T16:33:54Z

Dave, Here is the nabu PR with the addition of a cfgURL option.

so like

nabu release --cfgURL https://example.org/data/nabuconfig.yaml --prefix summoned/dataverse --endpoint localoxi

Also in the PR (sorry for overloading, I did it all in the same branch) is the "datacatalog" bit where I put in a datacatalog in the release graphs with all the named names in it.

feel free to push back for changes improvements...

example

<https://gleaner.io/xid/genid/csrn7l7g8s2n3pqu8pbg> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://schema.org/DataCatalog> .
<https://gleaner.io/xid/genid/csrn7l7g8s2n3pqu8pbg> <https://schema.org/dateCreated> "2024-11-15 10:16:20" . 
<https://gleaner.io/xid/genid/csrn7l7g8s2n3pqu8pbg> <https://schema.org/description> "GleanerIO Nabu generated catalog" .
<https://gleaner.io/xid/genid/csrn7l7g8s2n3pqu8pbg> <https://schema.org/provider> <https://gleaner.io/xid/genid/csrn7l7g8s2n3pqu8pc0> .
<https://gleaner.io/xid/genid/csrn7l7g8s2n3pqu8pbg> <https://schema.org/publisher> <https://gleaner.io/xid/genid/csrn7l7g8s2n3pqu8pcg> .
<https://gleaner.io/xid/genid/csrn7l7g8s2n3pqu8pc0> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://schema.org/Organization> .
<https://gleaner.io/xid/genid/csrn7l7g8s2n3pqu8pc0> <https://schema.org/name> "africaioc" .
<https://gleaner.io/xid/genid/csrn7l7g8s2n3pqu8pcg> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://schema.org/Organization> .
<https://gleaner.io/xid/genid/csrn7l7g8s2n3pqu8pcg> <https://schema.org/name> "gleaner.oih" .<https://gleaner.io/xid/genid/csrn7l7g8s2n3pqu8pbg> <https://schema.org/dataset> <urn:gleaner.io:doos:africaioc:data:000db5913f586b6d06bc0e3f59f33e299878e9f1> .
<https://gleaner.io/xid/genid/csrn7l7g8s2n3pqu8pbg> <https://schema.org/dataset> <urn:gleaner.io:doos:africaioc:data:00f8c810e243e27e3ee5f5f06bc792e1b4b4aec5> .
<https://gleaner.io/xid/genid/csrn7l7g8s2n3pqu8pbg> <https://schema.org/dataset> <urn:gleaner.io:doos:africaioc:data:01db27cf260cd84f0d5074748070bde443c43985> .
...

Added support for loading configuration files from a URL and improved data processing by implementing comprehensive Skolemization and graph association. Enhanced bulkLoader with a new flag for archiving, updated documentation, and incremented the version.

Replace standard log package with logrus for enhanced logging capabilities across the project. Update RDF metadata generation to include dynamic timestamps and bucket-derived names, improving the accuracy and relevance of generated data descriptions.

valentinedwv

Lets work to avoid blind nodes, since we control the datacatalog generation. and use our own urns.

valentinedwv · 2024-11-15T18:13:00Z

internal/objects/pipecopy.go

+		// Once we are done with the loop, put in the triples to associate all the graphURIs with the org.
+		if lastProcessed {
+
+			data := `_:b0 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://schema.org/DataCatalog> .


Why not make this a urn?
urn:gleaner.io:{SOURCE}:datacatalog

Avoid dangling triples.

valentinedwv · 2024-11-15T18:22:57Z

internal/objects/pipecopy.go

+			data := `_:b0 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://schema.org/DataCatalog> .
+_:b0 <https://schema.org/dateCreated> "` + time.Now().Format("2006-01-02 15:04:05") + `" . 
+_:b0 <https://schema.org/description> "GleanerIO Nabu generated catalog" .
+_:b0 <https://schema.org/provider> _:b1 .


same here...
urn:gleaner.io:{IMPLNET}:provider

valentinedwv · 2024-11-15T18:23:51Z

internal/objects/pipecopy.go

+_:b0 <https://schema.org/dateCreated> "` + time.Now().Format("2006-01-02 15:04:05") + `" . 
+_:b0 <https://schema.org/description> "GleanerIO Nabu generated catalog" .
+_:b0 <https://schema.org/provider> _:b1 .
+_:b0 <https://schema.org/publisher> _:b2 .


urn:gleaner.io:{source}:organization

valentinedwv · 2024-11-15T18:26:23Z

docs/README.md

+Nabu can also read the configuration file from over the network
+
+```
+go run ../../cmd/nabu/main.go release --cfgURL https://provisium.io/data/nabuconfig.yaml --prefix summoned/dataverse --endpoint localoxi


nabu release --cfgURL https://provisium.io/data/nabuconfig.yaml --prefix summoned/dataverse --endpoint localoxi

fils · 2024-11-15T19:13:50Z

I can make those changes, but the step at line 145

nabu/internal/objects/pipecopy.go

Lines 144 to 146 in c897b6f

    
           // TODO:  Skolemize with sdataWithContext 
        
           sdata, err := graph.Skolemization(data, "release graph prov for ORG") 
        
           if err != nil {

does Skolemize this and remove all blank nodes. It doesn't make a formal URN like your suggestion does though.

In my approach the SPARQL would need to look for type DataCatalog with a schema:provider scheme:publisher property values text.

In yours it would be the same but with by a URN/IRI.

Happy to make the change though.

I need to pull the IMPLNET... which is nicer than the bucket name.

valentinedwv · 2024-11-15T19:56:24Z

I can make those changes, but the step at line 145

nabu/internal/objects/pipecopy.go

Lines 144 to 146 in c897b6f

// TODO: Skolemize with sdataWithContext

sdata, err := graph.Skolemization(data, "release graph prov for ORG")

if err != nil {

does Skolemize this and remove all blank nodes. It doesn't make a formal URN like your suggestion does though.

In my approach the SPARQL would need to look for type DataCatalog with a schema:provider scheme:publisher property values text.

In yours it would be the same but with by a URN/IRI.

Happy to make the change though.

I need to pull the IMPLNET... which is nicer than the bucket name.

Yes, just looking for consistency, rather than randomness. Also, makes it easier to find the exact catalog for a source if it has a consistent ID, which is what we want to do.

Enhanced the PipeCopy function to include logging of organization name and generation of named graphs with unique URIs for RDF datasets based on organization names. Included a helper function to generate date-based SHA256 hashes to ensure unique graph URIs.

fils · 2024-11-15T20:05:01Z

@valentinedwv

here is what I have now. Noticed these were triples, not quads, so fixed that.

IRI for catalog:  <urn:gleaner.io:doos:datacatalog>
IRI for publisher:  <urn:gleaner.io:africaioc:publisher>
IRI for provider: <urn:gleaner.io:doos:provider>
IRI for named graph for these triples:  <urn:gleaner.io:doos:africaioc:datacatalog:85856a8f3f88628e4e70450c623edfaf7a071eacf721d78115a742071a8d01d0>

<urn:gleaner.io:doos:datacatalog> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://schema.org/DataCatalog> <urn:gleaner.io:doos:africaioc:datacatalog:85856a8f3f88628e4e70450c623edfaf7a071eacf721d78115a742071a8d01d0> .
<urn:gleaner.io:doos:datacatalog> <https://schema.org/dateCreated> "2024-11-15 14:03:37" <urn:gleaner.io:doos:africaioc:datacatalog:85856a8f3f88628e4e70450c623edfaf7a071eacf721d78115a742071a8d01d0> .
<urn:gleaner.io:doos:datacatalog> <https://schema.org/description> "GleanerIO Nabu generated catalog" <urn:gleaner.io:doos:africaioc:datacatalog:85856a8f3f88628e4e70450c623edfaf7a071eacf721d78115a742071a8d01d0> .
<urn:gleaner.io:doos:datacatalog> <https://schema.org/provider> <urn:gleaner.io:doos:provider> <urn:gleaner.io:doos:africaioc:datacatalog:85856a8f3f88628e4e70450c623edfaf7a071eacf721d78115a742071a8d01d0> .
<urn:gleaner.io:doos:datacatalog> <https://schema.org/publisher> <urn:gleaner.io:africaioc:publisher> <urn:gleaner.io:doos:africaioc:datacatalog:85856a8f3f88628e4e70450c623edfaf7a071eacf721d78115a742071a8d01d0> .
<urn:gleaner.io:doos:provider> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://schema.org/Organization> <urn:gleaner.io:doos:africaioc:datacatalog:85856a8f3f88628e4e70450c623edfaf7a071eacf721d78115a742071a8d01d0> .
<urn:gleaner.io:doos:provider> <https://schema.org/name> "doos" <urn:gleaner.io:doos:africaioc:datacatalog:85856a8f3f88628e4e70450c623edfaf7a071eacf721d78115a742071a8d01d0> .
<urn:gleaner.io:africaioc:publisher> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://schema.org/Organization> <urn:gleaner.io:doos:africaioc:datacatalog:85856a8f3f88628e4e70450c623edfaf7a071eacf721d78115a742071a8d01d0> .
<urn:gleaner.io:africaioc:publisher> <https://schema.org/name> "africaioc" <urn:gleaner.io:doos:africaioc:datacatalog:85856a8f3f88628e4e70450c623edfaf7a071eacf721d78115a742071a8d01d0> .
<urn:gleaner.io:doos:datacatalog> <https://schema.org/dataset> <urn:gleaner.io:doos:africaioc:data:000db5913f586b6d06bc0e3f59f33e299878e9f1> <urn:gleaner.io:doos:africaioc:datacatalog:85856a8f3f88628e4e70450c623edfaf7a071eacf721d78115a742071a8d01d0> .
<urn:gleaner.io:doos:datacatalog> <https://schema.org/dataset> <urn:gleaner.io:doos:africaioc:data:00f8c810e243e27e3ee5f5f06bc792e1b4b4aec5> <urn:gleaner.io:doos:africaioc:datacatalog:85856a8f3f88628e4e70450c623edfaf7a071eacf721d78115a742071a8d01d0> .
<urn:gleaner.io:doos:datacatalog> <https://schema.org/dataset> <urn:gleaner.io:doos:africaioc:data:01db27cf260cd84f0d5074748070bde443c43985> <urn:gleaner.io:doos:africaioc:datacatalog:85856a8f3f88628e4e70450c623edfaf7a071eacf721d78115a742071a8d01d0> .

valentinedwv

Think pattern needs to be
urn:gleaner.io:ORG:SOURCE:object

valentinedwv · 2024-11-15T21:01:11Z

internal/objects/pipecopy.go

+		if lastProcessed {
+
+			data := `<urn:gleaner.io:` + orgname + `:datacatalog> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://schema.org/DataCatalog> .
+<urn:gleaner.io:` + orgname + `:datacatalog> <https://schema.org/description> "GleanerIO Nabu generated catalog" .


urn:gleaner.io:ORG:SOURCE:datacatalog

valentinedwv · 2024-11-15T21:01:38Z

internal/objects/pipecopy.go

+<urn:gleaner.io:` + orgname + `:datacatalog> <https://schema.org/publisher> <urn:gleaner.io:` + getLastElement(prefix) + `:publisher> .
+<urn:gleaner.io:` + orgname + `:provider> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://schema.org/Organization> .
+<urn:gleaner.io:` + orgname + `:provider> <https://schema.org/name> "` + orgname + `" .
+<urn:gleaner.io:` + getLastElement(prefix) + `:publisher> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://schema.org/Organization> .


urn:gleaner.io:ORG:SOURCE:publisher

valentinedwv · 2024-11-15T21:02:26Z

internal/objects/pipecopy.go

+<urn:gleaner.io:` + orgname + `:datacatalog> <https://schema.org/provider> <urn:gleaner.io:` + orgname + `:provider> .
+<urn:gleaner.io:` + orgname + `:datacatalog> <https://schema.org/publisher> <urn:gleaner.io:` + getLastElement(prefix) + `:publisher> .
+<urn:gleaner.io:` + orgname + `:provider> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://schema.org/Organization> .
+<urn:gleaner.io:` + orgname + `:provider> <https://schema.org/name> "` + orgname + `" .


Since there is one provider, this is good
urn:gleaner.io:eco:provider

fils added 4 commits December 4, 2023 10:29

updates to function.go to work with Oxigraph

59c259b

Merge branch 'refs/heads/dev' into df-dev

98053e1

fils requested a review from valentinedwv November 15, 2024 16:33

valentinedwv requested changes Nov 15, 2024

View reviewed changes

fils requested a review from valentinedwv November 15, 2024 20:05

valentinedwv approved these changes Nov 15, 2024

View reviewed changes

valentinedwv requested changes Nov 15, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Df dev with cfgURL and datacatalog in release graph #57

Df dev with cfgURL and datacatalog in release graph #57

fils commented Nov 15, 2024

valentinedwv left a comment

valentinedwv Nov 15, 2024

valentinedwv Nov 15, 2024

valentinedwv Nov 15, 2024

valentinedwv Nov 15, 2024

fils commented Nov 15, 2024

valentinedwv commented Nov 15, 2024

fils commented Nov 15, 2024

valentinedwv left a comment

valentinedwv Nov 15, 2024

valentinedwv Nov 15, 2024

valentinedwv Nov 15, 2024

Df dev with cfgURL and datacatalog in release graph #57

Are you sure you want to change the base?

Df dev with cfgURL and datacatalog in release graph #57

Conversation

fils commented Nov 15, 2024

valentinedwv left a comment

Choose a reason for hiding this comment

valentinedwv Nov 15, 2024

Choose a reason for hiding this comment

valentinedwv Nov 15, 2024

Choose a reason for hiding this comment

valentinedwv Nov 15, 2024

Choose a reason for hiding this comment

valentinedwv Nov 15, 2024

Choose a reason for hiding this comment

fils commented Nov 15, 2024

valentinedwv commented Nov 15, 2024

fils commented Nov 15, 2024

valentinedwv left a comment

Choose a reason for hiding this comment

valentinedwv Nov 15, 2024

Choose a reason for hiding this comment

valentinedwv Nov 15, 2024

Choose a reason for hiding this comment

valentinedwv Nov 15, 2024

Choose a reason for hiding this comment