Generic LDNodes and RO-Crate parsing rework #496

HLWeil · 2025-02-27T17:25:34Z

Reworked Generic JSON-LD Objects

This is a generic JSON-LD implementation that represents those aspects of the JSON-LD specification which are relevant for handling ARC-RO-Crate:

Datamodel
- LDContext (Contains mappings from compacted terms to extended IRIs)
- LDNode (Our basic objects with @id, types and fields)
- LDGraph (Basically a collection of nodes; Top-level object in the RO-Crate)
- LDRef (Object containing only @id, meant for linking)
- LDValue Expanded, self-contextualizing values
Algorithms
- Flatten
- Compact
- WIP Unflatten
Generic JSON encoders and decoders

Added ARC-RO-Crate profile layer

This layer acts like a ARC-RO-Crate view on the generic JSON-LD model. All actual handled objects are either FSharp primitive types or generic JSON-LD objects. Through making use of static methods which in turn use static IRIs, a loose typing is emulated. These static IRI mostly consist of types and properties from schema.org and bioschemas.org.

Added a set of static classes, containing functions for
- getting properties
- setting properties
- generating default @ids
- validating given LDNode against requirements for this type
- create functions for LDNodes matching the requiremenets for this type
For the following types/profiles:
- Comment
- DefinedTerm
- PropertyValue
- PostalAddress
- Organization
- Person
- ScholarlyArticle
- Sample
- File
- LabProtocol
- LabProcess
- Dataset
Added contexts of RO-Crate specification 1.1 and 1.2DRAFT
Added context for all other fields exclusive to the ARC-RO-Crate profile

ARC Scaffold <-> ARC-RO-Crate conversion layer

Added functions for Conversion between Scaffold and RO-Crate representation types.
E.g.

Ontology Annotation <-> DefinedTerm
ArcTable <-> LabProcess Sequence
ArcInvestigation <-> Dataset

Added top-level members to ARC Scaffold object to parse to and from an RO-Crate JSON-LD metadata string.

To-do:

Top level parsers
DOI and PubmedID as identifier
Bioschemas context
Some @id are empty (LabProtocol)
Top-level roundabout tests
Link datafiles instead of fragment selectors from datasets

To-do after this PR is merged:

Datamap support
Make use of @list for ordered process inputs and outputs
CI job, running the RO-Crate output for ARC(Prototype?) against RO-Crate validator

closes #326
closes #400
closes #429
closes #434
closes #465
closes #494
closes #495

- Json: parse `@type`/`additionalType` from either string (-> singleton) or array field, always write array - add tests

HLWeil · 2025-02-27T23:43:57Z

@floWetzels could you also take a look onto whether #326 is actually fixed or not (e.g. by inspecting a written RO-Crate metadata json)

floWetzels · 2025-02-28T15:35:59Z

I tested the metadata json file generated for the ArcPrototype with the rocrate-validator. I found the following errors that need to be fixed:

Studies und assays have wrong ids - the path to the isa-xlsx file. It should be the path to the directory ending with a /.
Studies and assays are missing the hasPart property
The root data entity has no license
The root data entity has no datePublished

Furthermore, I noticed the following when parsing the file manually:

Paths must be relative to the root of the ro-crate
Ad-hoc ids (e.g. for processes, paramters, parameter values) should begin with a # symbol
The id of PropertyValue objects looks weird to me. It contains the propertyID, which could be a potential source for errors. Previously, it was generated only from name and value, if I am not mistaken.

floWetzels · 2025-03-04T08:29:32Z

Looks a lot better now! The validator accepts the RO-Crate. When looking through the file, I noticed a few more things:

Some @id values contain spaces. This appears almost throughout all types, e.g. Organization, PropertyValue, Process, Protocol, Sample etc. This is not necessarily a problem, the json-ld playground is able to handle it for example. It is also not completely avoidable, since file paths can contain spaces. I would prefer to replace them in ad-hoc ids though. The old export did so with a simple replace statement.
For some types, ad-hoc ids are generated although a URI or similar identifier exists. I found two examples in the ArcPrototype, DefinedTerm and Person. A DefinedTerm should use its termCode as id, if it exists. For Person, I found an instance where an email address exists, but an ad-hoc id was assigned.
Some termCode entries of DefinedTerm contain multiple URIs separated by an underscore. This should not be the case, right? Here's an example: "termCode": "http://purl.obolibrary.org/obo/scoro_http://purl.org/spar/scoro/principal-investigator".

HLWeil · 2025-03-04T10:01:09Z

I incorporated your suggested fixes and included the test file in d3d786e

kMutagene · 2025-03-06T13:11:42Z

@HLWeil i think adding a test that performs the validation of the ro-crate file that @floWetzels did manually is needed. If not in this PR then via separate issue

HLWeil · 2025-03-06T13:29:37Z

Yes!

I included it in the "to-do after PR" so I'll open up the issues when I merge this

kMutagene

See comments

kMutagene · 2025-03-06T13:27:29Z

src/ROCrate/Generic/File.fs

+    static member genId(name : string) =
+        $"{name}"
+
+    static member validate(dt : LDNode, ?context : LDContext) =


Is this the equivalent of the missing file profile for nfdi4plants/isa-ro-crate-profile#57?

... I mean yesn't.

It isn't missing, File = Data

kMutagene · 2025-03-06T13:28:25Z

src/ROCrate/Generic/Organization.fs

Is this the missing organization profile for nfdi4plants/isa-ro-crate-profile#57 ?

kMutagene · 2025-03-06T13:28:49Z

src/ROCrate/Generic/PostalAddress.fs

Is this the missing postaladdress profile for nfdi4plants/isa-ro-crate-profile#57 ? If we have 0 mandatory fields, maybe we just don't need to support this type?

Well there were discussions about it for the specification. Also requests for the implementation: #465

I currently handle it in ARC Scaffold by just writing the json blob in the string field.

kMutagene · 2025-03-06T13:32:19Z

src/ROCrate/Generic/PropertyValue.fs

+    static member setValueReference(pv : LDNode, valueReference : string, ?context : LDContext) =
+        pv.SetProperty(PropertyValue.valueReference, valueReference, ?context = context)
+
+    static member validate(pv : LDNode, ?context : LDContext) =


Is this missing validate functions for PropertyValue - DOI and PropertyValue - PubMedID ?

There's currently not a validate function for doi and pubmedID.

The convertes can be found here:

ARCtrl/src/ARCtrl/Conversion.fs

Line 1015 in b755341

static member tryDecomposeDOI (doi : LDNode, ?context : LDContext) : string option =

kMutagene · 2025-03-06T13:33:26Z

src/ROCrate/Generic/ScholarlyArticle.fs

+    static member setAuthors(s : LDNode, authors : ResizeArray<LDNode>, ?context : LDContext) =
+        s.SetProperty(ScholarlyArticle.author, authors, ?context = context)
+
+    static member tryGetUrl(s : LDNode, ?context : LDContext) =


We dropped url as property of this profile

kMutagene · 2025-03-06T13:34:46Z

src/ROCrate/ISAProfile/Assay.fs

Why are you deleting all these types?

That was a misunderstanding between me, myself and I

kMutagene · 2025-03-06T13:39:56Z

tests/ROCrate/ISAProfile/Person.Tests.fs

@@ -14,7 +14,7 @@ let mandatory_properties = Person(
 let all_properties = Person(


How are these tests still passing when you deleted all the types tested? maybe not seeing the forest for the trees here

kMutagene · 2025-03-06T13:40:57Z

tests/ROCrate/Main.fs

-    Tests.ScholarlyArticle.main
+    Tests.LDContext.main
+    Tests.LDNode.main
+    //Tests.Dataset.main


hmm i think i found the answer, you are not running the tests. Why fix them then?

…rlyArticle converters

HLWeil · 2025-03-07T16:34:46Z

Alright the comments should be resolved.

I reincluded the initial ROCrate Types, and renamed the static classes using the LD prefix.
LDPropertyValue now has DOI and PubMedID helper functions
Implemented and tested DOI and PubMedID backwards compatability (using url and sameAs)

kMutagene and others added 30 commits February 10, 2025 15:30

Add LDObject.tryFromDynamicObj + tests

b434ade

LDObject: model @type/additionalType as arrays:

e5efa2f

- Json: parse `@type`/`additionalType` from either string (-> singleton) or array field, always write array - add tests

Fix running single test projects

556adb3

some LDObject parsing fixes

10ab9a3

wip LDObject from dynamic obj

0eab82a

update DynamicObj dependency and fix for LDObject from DynamicObj

7de9bdf

fix LDObject from DynamicObj function

0ad4df6

add basic ld context resolving

bc55487

add ro-crate 1.1 context

015410a

make ldcontext term resolution recursive

c69106a

split up LDObject into LDNode, LDRef and LDValue

e7a8c6c

add basic LD form algorithms

7b62486

start working on basic generic LDNode access classes

4222bc3

add basic LDContext tests

664b86b

add context copy tests

bda9098

start working on ldnode tests

bd11928

add basic ldnode tests

1c43c6d

some fixes for ldnodes according to tests

1ca9ac9

add ldnode compact and flatten tests

27db166

start on ldnode json io

477f921

add various ldobject json parsing tests

030c9c3

add basic ldgraph test and context parsing hotfixes fixes

c9c6020

add some static types from isa-ro-crate profile

48efd4f

add labprocess and labprotocol rocrate static classes

59edad3

start working on scaffold RO-Crate type conversion

05d5ea6

finish up preliminary version of Scaffold ROCrate Process conversion

fdb4f16

add and update first bulk of RO-Crate Scaffold conversion tests

633438c

some LabProcess Table conversion fixes

90647e0

fix remaining LabProcess ArcTable conversion errors

e1dc2cb

start working on top level RO Crate types and conversions

f18abad

HLWeil added 2 commits February 27, 2025 23:56

add doi and pubmedid identifier handling

d43845e

fix js and py

b239c35

This was referenced Feb 27, 2025

[Feature Request] Test Cases for RO-Crate Export with Fixed Output #487

Open

handle people address from ro-crate json #465

Closed

HLWeil added 4 commits March 3, 2025 11:29

add defaults for requried RO-Crate fields

3fa2701

small changes to generated @ids in ro-crate

1358454

first version of making data file paths absolute to ARC-Root

e392461

add behaviour to fill assay and study hasParts with datafiles

0e9a4d7

HLWeil added 3 commits March 4, 2025 10:27

various fixes to improve @ids in Json-LD writing

308098f

add new ArcPrototype RO-Crate testfiles

d3d786e

small fix to rocrate conversion test

23d38dd

HLWeil marked this pull request as ready for review March 4, 2025 12:04

HLWeil added 4 commits March 4, 2025 14:23

extend LDContext functionality by compactIRI reverse resolving

6e3e1fd

add fromDeprecatedRoCrate function and some fixes

1e3ad23

add ro-crate parsing tests

6cf8105

various fixes for reading new and deprecated ARC RO-Crates

b755341

kMutagene requested changes Mar 6, 2025

View reviewed changes

HLWeil added 2 commits March 7, 2025 14:28

reinclude isaprofile types and prefix static class names with LD

f997440

added doi and pubmed helper functions to LDPropertyValue and LDSchola…

0d82e3c

…rlyArticle converters

kMutagene mentioned this pull request Mar 10, 2025

Improve LDObject, add ISA model API #478

Closed

kMutagene approved these changes Mar 10, 2025

View reviewed changes

HLWeil merged commit b3d1537 into main Mar 10, 2025
2 checks passed

HLWeil deleted the ld-object-context branch March 10, 2025 12:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generic LDNodes and RO-Crate parsing rework #496

Generic LDNodes and RO-Crate parsing rework #496

HLWeil commented Feb 27, 2025 •

edited

Loading

HLWeil commented Feb 27, 2025

floWetzels commented Feb 28, 2025

floWetzels commented Mar 4, 2025

HLWeil commented Mar 4, 2025

kMutagene commented Mar 6, 2025

HLWeil commented Mar 6, 2025

kMutagene left a comment

kMutagene Mar 6, 2025

HLWeil Mar 7, 2025

HLWeil Mar 7, 2025

kMutagene Mar 6, 2025

HLWeil Mar 7, 2025

kMutagene Mar 6, 2025

HLWeil Mar 7, 2025

kMutagene Mar 6, 2025

HLWeil Mar 7, 2025

kMutagene Mar 6, 2025 •

edited

Loading

kMutagene Mar 6, 2025

HLWeil Mar 7, 2025

kMutagene Mar 6, 2025

kMutagene Mar 6, 2025 •

edited

Loading

HLWeil commented Mar 7, 2025

		@@ -14,7 +14,7 @@ let mandatory_properties = Person(
		let all_properties = Person(

Generic LDNodes and RO-Crate parsing rework #496

Generic LDNodes and RO-Crate parsing rework #496

Conversation

HLWeil commented Feb 27, 2025 • edited Loading

Reworked Generic JSON-LD Objects

Added ARC-RO-Crate profile layer

ARC Scaffold <-> ARC-RO-Crate conversion layer

HLWeil commented Feb 27, 2025

floWetzels commented Feb 28, 2025

floWetzels commented Mar 4, 2025

HLWeil commented Mar 4, 2025

kMutagene commented Mar 6, 2025

HLWeil commented Mar 6, 2025

kMutagene left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kMutagene Mar 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kMutagene Mar 6, 2025 • edited Loading

Choose a reason for hiding this comment

HLWeil commented Mar 7, 2025

HLWeil commented Feb 27, 2025 •

edited

Loading

kMutagene Mar 6, 2025 •

edited

Loading

kMutagene Mar 6, 2025 •

edited

Loading