Prerelease Automated Validation Pipeline for Jargon Artifacts

[ashleythedeveloper]

Currently, we're encountering issues where invalid artifacts (JSON-LD context files) are released from Jargon without proper validation. These issues include conflicting protected terms that are only discovered after release. This leads to implementation problems downstream and requires additional releases to fix them. To prevent these problems and improve the quality of our releases, we need a systematic approach to catching these validation errors before artifacts are released.

An example of this can be found here, where we are redefining protected terms within the Digital Conformity Credential.

Thank you @PatStLouis for picking up on this.

After a few conversations, I've been investigating ways to validate the artifacts produced by Jargon (JSON-LD context files, schemas, and sample credentials) and integrate this validation into our prerelease workflow. This proposal aims to prevent the release of invalid artifacts by implementing checks before release.

Based on my initial analysis, we can catch most of the errors by implementing a GitHub Actions pipeline that executes a series of validation test cases using libraries like Ajv and jsonld.js. This pipeline would be triggered on a snapshot within Jargon via a GitHub webhook, running validation checks against the generated artifacts.

Proposed Artifact Testing Pipeline:

flowchart TD
    A[Jargon Snapshot Created] -->|GitHub Webhook| B[Trigger GitHub Actions Pipeline]
    B --> C{Run Validation Tests}
    
    C -->|Validate| D[Context Tests]
    C -->|Validate| E[Schema Tests]
    C -->|Validate| F[Credential Tests]
    
    D & E & F --> G{All Tests Pass?}
    
    G -->|No| H[Report Failures]
    H --> I[Manual Review Required]
    I --> B
    
    G -->|Yes| J[Validation Complete]
    J --> K[Proceed to Release]

Loading

What would be tested:

Context

• JSON validity
• Conformance to JSON-LD syntax rules
• Resolvability of remote contexts (if present)
• Absence of conflicting protected terms

Schema

• JSON validity
• Correct meta-schema usage (2020-12)
• Meta-schema conformance

Sample Credential

• JSON validity
• Conformance to the corresponding UNTP schema
• Conformance to the W3C v2 VCDM schema
• Context resolvability
• Absence of conflicting protected terms
• Document expandability
• Absence of undefined terms

To catch current errors, particularly redefined protected terms, in this case, testing the context file alone is insufficient. These errors often emerge when multiple types are assigned and those types have conflicting terms (see @PatStLouis example above). Therefore, we need to test the sample credentials produced by Jargon alongside the context files it generates.

Additionally, we need to ensure that implementers can extend a given UNTP data model while maintaining conformance to the core specification. Based on initial analysis, automating this validation is challenging without prior knowledge of the credentials and updated test cases. For now, this validation will likely need to be a manual pre-release task. However, I welcome suggestions for potential automation approaches.

Moving forward, we also plan to incorporate many of these test cases into the UNTP Playground, where implementers can validate their credentials upon upload.

[absoludity]

Nice write-up Ash. After a deeper look below (though I'll preface with I'm new to a lot of this so still learning context), my summary is, yes having such a test to ensure we don't publish invalid LDs or Schemas is of course needed, and the outline you've done is mostly fine, but it won't catch valid yet meaningless or confusing output that may still cause problems. My main comment is that I do not think we should be publishing artifacts directly from jargon automatically, even if your new test passes. The pipeline for publishing these generated artifacts should always include requiring a human to review the changes between the previous version and the to-be-published version - not only when the test fails - and storing the approved version in git. This is for two reasons:

1. It's a 3rd party product that has its own release cycles which may change the expected output without our knowledge (ie., the 0.5.1 artifact may have unexpected differences from the 0.5.0 artifact, in addition to the expected ones), and
2. It's important that these files are a simple as possible (which I don't think the generated artifacts currently are, more below) so that they can be understood by devs (us and downstream) to detect issues and be built upon.

That's what I've come away with from trying to understand the problem more deeply. I don't know the jargon tool well, so please correct any mistakes below :) Now for the long version...

The problem - UX != code

For me, before defining a solution, I really want to properly understand the problem. You mention that

we're encountering issues where invalid artifacts (JSON-LD context files) are released from Jargon without proper validation

but what exactly is the issue? @PatStLouis mentions the fact that all the core properties are repeated on the dcc extended entity, but I'm not sure that (alone) is itself causing the doc to be invalid (identical redefinitions are not invalid), or that it's the only source of confusion (not only are the properties repeated, but many whole untp-core entities are redefined identically in the dcc context as well).

Possible non-issue: identical redefinitions of properties

Using the example above, if we look at the ConformityAttestation's definition from the 0.5.0 DCC LD:

    "ConformityAttestation": {
      "@protected": true,
      "@id": "untp-dcc:ConformityAttestation",
      "@context": {
        "assessorLevel": "untp-core:assessorLevel",
        "assessmentLevel": "untp-core:assessmentLevel",
        "attestationType": "untp-core:attestationType",
        "issuedToParty": {
          "@id": "untp-dcc:issuedToParty",
          "@type": "@id"
        },
        "authorisation": "untp-core:authorisation",
        "conformityCertificate": "untp-core:conformityCertificate",
        "auditableEvidence": "untp-core:auditableEvidence",
        "scope": {
          "@id": "untp-dcc:scope",
          "@type": "@id"
        },
        "assessment": {
          "@id": "untp-dcc:assessment",
          "@type": "@id"
        }
      }
    },

we see that it is mostly redefining all the untp-core.Attestation properties with their identical untp-core counterparts - their original definition. AFAICT, this is actually allowed. The spec says the following for protected terms:

While protected terms can in general not be overridden, there are two exceptions to this rule. The first exception is that a context is allowed to redefine a protected term if the new definition is identical to the protected term definition

So although it may not be best practise, that part of the output may not be an issue even if it is not ideal (though testing with the tools you mentioned will verify, I guess).

Confusing non-issue: identical redefinitions of whole entities

Another oddity (to my unfamiliar context) is that many of the actual core entities are redefined (identically) in the DCC LD (and probably all the others too, haven't checked). If you search through the 0.5.0 dcc LD you'll find the core Product, the core Attestation, Declaration etc., all redefined identically to their core definitions. It appears to be all the entities which are extended in the UX - but I don't understand why they are re-defined in DCC context file (they don't need to be - that's the whole point of linked documents right?) - might be something to follow up with @colugo ?

Even though this may not be an issue (see identical redefinition above), I think it is indirectly causing the actual issues, such as...

Issue: non-identical redefinition

What is an issue is that the ConformityAttestation.issuedToParty term is redefined differently - see above. jargon.sh actually highlights this quite helpfully in the UX as well, if I open the editor for the DCC (see bottom left):

Do we know why it is redefined differently here ? Looking at the data model in jargon makes me wonder whether it's either

1. Because the Party entity in DCC has some fields unset (such as registrationCountry)? The UI (above) makes it seem that way, as if the DCC Party is a new entity extended from the core Party, but in the DCC context output only includes the identical redefinition of the core Party, with the normal fields (registrationCountry etc.)
2. Because we wanted a new @id for the field (and missing that it means an invalid redefinition, or perhaps before @protected was a thing), or
3. Because it was actually a mistake or input error (drag-and-drop, or click or selection error or similar) and we've referenced the identically redefined dcc Party rather than the core Party entity for the issuedToParty property (all the other identical properties on ConformityAttestation refer directly to the similar core property, both on the jargon UX and the LD file).

Similarly, I'm not sure why ConformityAssessment's assessedProduct and assessedFacility (which are new fields) need to refer to the identical copy of Facility and Product in the DCC LD, rather than directly to the UNTP core ones - but perhaps that's more related to the fields defining new ids - I'm not sure.

What is, I suspect, an issue though we may not have seen it yet, is that ConformityAssessment.assessmentDate does get redefined with the id referring to a dcc assessment date, rather than the core one. Again, it's not clear to me why this would be needed - perhaps it's something I'm not yet understanding (and this isn't highlighted by the jargon UX, though I may have misunderstood its validity - again, playing with the tools you suggested should prove it).

Anyway, once the issue is understood more deeply (not sure that I have yet), I think it's easier to focus just on what needs to change to be valid. If it were me, I'd play with the tools you've suggested and test out the above to see exactly what is valid and invalid, but importantly, even if an automated test finds the jargon-generated updated artifacts valid, the workflow should be producing a diff of changes for a human to verify the changes are the expected ones, rather than publishing without a human looking verifying it's not potentially including unnecessary redefinitions or similar that may still be valid but making it harder to use.

[absoludity]

Regarding the invalid redefinition of issuedToParty, after learning a bit about jargon an how to use it, I see that yes, the Party entity in DCC appears to be explicitly subclassed to remove those extra fields and then used as a replacement for untp.Party in the ConformityAttestation, which causes the invalid redefinition.

As a suggestion (only), I've created a jargon suggestion which updates the properties in the DCC that refer to the sub-classed Party to instead use untp.Party (ie. it no longer has the illegal redefinition, but the Party property used will have the extra fields as it's a vanilla untp.Party.

https://jargon.sh/user/unece/ConformityCredential/suggestion/1/?view=changes