How to handle property result in lineage

[pmaria]

The problem

When building lineage in the current way, we represent source or orchestrated properties in the following structure:

{
  "subject": {
    // Object reference
  },
  "property": ... // name of the property
  "value": ... // value of the property
}

There are several challenges here.

Challenge 1: Unknown schema of key properties of object references

In the object reference, we cannot predict the schema of the identifying properties.

The referenced object could have one or more identifying properties. So it is also not possible to map to a single object key.

Currently we have a temporary "solution" which only works for object types with a single identifying property. Here we extract the value of the identifying property and return that as the value of the objectKey in the ObjectReference object. For example;

{
  "subject": {
     "objectKey": "12345",
     "objectType: "Foo"
  },
  "property": ... // name of the property
  "value": ... // value of the property
}

This is not future-proof

Challenge 2: Value type variability

When returning a value for a source or orchestrated property, we cannot know what the value type will be. This could be e.g. one of the scalar types, or an object type, or something else.

For this we currently have a solution that introduces an intermediary node which contains properties named Xvalue where X indicates the type of value. For example:

{
  "subject": {
    "objectKey": "12345",
    "type": "Foo"
  },
  "property": "bar",
  "value": {
    "stringValue": null,
    "booleanValue": true,
    "integerValue": null,
    "objectValue": null,
    ...
  }
}

This is obviously not an ideal solution. And furthermore, the objectValue suffers from the same issue as described in Challenge 1, in that the schema of the referenced object cannot be known upfront.

Possible solutions

Not really sure. We could look into using GraphQl's Custom scalars.
This would however be a burden on clients. That being said, so is the current approach.