Clowder Metadata design

[longshuicy]

the structure of metadata definition could look like below

{
    "name" : "Unit",
    "description" : "Unit",
    "context" : {
        "doi" : "https://schema.org/QuantitativeValue"
    },
    "fields" : [
        {
            "name" : "unit",
            "list" : false,
            "widgetType": "Select",
            "config": {
                 "type" : "enum",
                 "options": ["Ampere", "Kelvin", "Second"]
            },
            "required" : true
        }
    ]
}

Each field corresponding to an individual widget on the frontend.

A metadata entry could have multiple widgets (fields):

{
    "name" : "LatLon",
    "description" : "A set of Latitude/Longitude coordinates",
    "context" : {
        "longitude" : "https://schema.org/longitude",
        "latitude" : "https://schema.org/latitude"
    },
    "fields" : [
        {
            "name" : "longitude",
            "list" : false,
            "widgetType": "TextField",
            "config": {
                "type" : "float"
            },
            "required" : true
        },
        {
            "name" : "latitude",
            "list" : false,
            "widgetType": "TextField",
            "config": {
                "type" : "float"
            },
            "required" : true
        }
    ]
}

The content of config field can be flexible. It will be used to hold information that populates the frontend appearances, as well as validate the posted values. For example

   "config": {
                 "type" : "enum",
                 "options": ["Ampere", "Kelvin", "Second"]
            },

There will also be scenarios that a widget needs input of "undetermined number of input values". For example, picking as many points on a map. The usage of "list": true is designed for that. For example:

{
    "name" : "Coordinates",
    "description" : "Any number ofof Latitude/Longitude coordinates",
    "context" : {
        "point": "https://schema.org/point"
    },
    "fields" : [
        {
            "name" : "point",
            "list" : true,
            "widgetType": "Map",
            "config": {
                "type" : "tuple"
            },
            "required" : true
        }
    ]
}

And the correspondent posted metadata entry could look like:

{ 
    ...
    "context" : null, 
    "context_url" : null, 
    "definition" : "Coordinates", 
    "contents" : {
        "point" : [(111,112), (111, 113), (113, 114), (111, 115)]
    }, 
    "resource" : {
        "collection" : "files", 
        "resource_id" : ObjectId("62e16143e5f13ff2ff52f15d"), 
        "version" : NumberInt(1)
    }, 
    "agent" : {
        "id" : ObjectId("62e1614ce5f13ff2ff52f18a"), 
        "creator" : {
            "id" : ObjectId("62a9ed5514cd9901fa1828cf"), 
            "email" : "[email protected]", 
            "first_name" : "Chen", 
            "last_name" : "Wang"
        }, 
        "extractor" : null
    }, 
    "created" : ISODate("2022-07-27T16:01:16.252+0000"),
    ...
}

[longshuicy]

[max-zilla]

i really like this kind of structure. @ddey2 for our recent elasticsearch discussions, we could also refer to this definition for incoming metadata objects to check the data types so that elasticsearch doesn't have to guess as much (just one idea)

[max-zilla]

Follow-up from 10/28 discussion.

Part I - basic implementation

• metadata should be added to Elasticsearch mapping with data type object
• incoming metadata indexed with ES auto-determination of types
• include name of extractor in metadata object as well (to differentiate same field name in different extractors)
  • e.g. metadata.extractor.field1.field2 = 20
• See how facets UI can interact with

Part II - medium term

• can we better infer data types from context? very difficult. or, data types for widget if available?
• two facets for metadata:
  • METADATA FIELD - only need to include things below the extractor name (e.g. field1.field2) - once selected you can also pick operator (equals, less than) and value
  • METADATA SOURCE - list of extractors. tied to the FIELD selection (one filters the other one)

[max-zilla]

Some other ideas Luigi and I discussed:

• Store each metadata context in a Context collection with a unique ID. This includes metadata definition, or context URL, or or context JSON. (or convert MD Definition into this concept)
• we use context UUID in metadata path - "metadata.1234567.height" - to avoid issues with dot notation in the path.
• Users search for just metadata.height, and filter the extractor name separately if they want. behind the scenes we can replace that with 'metadata.*.height' or something if necessary to get same effect.
• Extractors referring to same context will not be distinguished in the index - they share the same mapping.