Type Metadata

[machaval]

Type Metadata

Currently DataWeave has the concept of metadata. Metadata is a collection of key values associated with a given value. This allows us to encode additional information such as class name in a java value, cdata in xml or any other stuff. The metadata values are used in three different contextx

1. Additional metadata injected by the reader
2. Hints to the writer to serialize the value correctly
3. Type Correcion hints

Metadata is also being specified in types. For example

type A = String {test: true}

In this example we are defining a new type A that represents the set of values that are String but also that has a metadata field test with value true.

This is useful for cases such as CData. For example in xml

<script>
   <![CDATA[
       Welcome to TutorialsPoint 
   ]]>
</script >

In this example this input is going to be read as

{
  script: "Welcome to TutorialsPoint"
}

But if we query the metadata properties of the script field using the ^ selector by doing payload.script.^ is going to return {cdata: true}

And we also have a type in Core called CData that is defined as
type CData = String {cdata: true}

Now with these two things we can validate if a String is CData or not by doing

payload.script is CData and this is because CData validate that the value is String and has the metadata cdata being set to true

This same pattern is being applied to many other things such as Enum in java etc.

Additionally to this use case it is also found in doing writter hints. And this is because currently the only way to attach a metadat property to a value is by using the type coercion operator as

output xml
---
{
  script: "Welcome to TutorialsPoint" as CData
}

By using this tecnique we are attaching {cdata:true} to the String "Welcome to TutorialsPoint". And later the XML writer will now that it needs to wrapp the text into CData. This is also consistent with one of our goals that is that we want the passthrough to work without a need. This means that if I use the XML as input and I do

output xml
---
{
  script: payload.script
}

It will continue to generate a CData xml element.

Types as Values

Now we also found that there is another usage of the Type Metadata. This usage is to store type information. This type information is use to encode additional type information but we don't want to use it a type constrain. For example when importing types from mule the types needs to be able to encode the TypeId. Or even a description or label from json schema

We can think of this a metadata of a Type Value and not as type constrains. So we may see it as

type T = String {cdata: true} as Type {label: "A T type"}

So in this case T is of type String with the constrain that it should have cdata equals true but also the Type value has metadata field called label.

So this way we could start encoding additional information when importing types from other type systems so that we can encode any additional missing information.

Like or Raw type

Another interesting view is that though having types metadat constrains are use full for some cases. For example if we implement the java type loader for example java!org::acme::MyCar::Class

package org.acme;
class MyCar {
  String brand;
  String plate;
}

and this MyCar type is being represented as

type Class = {
  brand:String,
  plate:String
} {class: "org.acme.MyClass"}

For this case we can then use it as

import java!org::acme::MyCar
---
{brand: "MercedesBenz", plate: "DW 020 FN"} as MyCar::Class

Now though this is very useful we also detected that sometimes we don't want to validate against that metadata information. For example in cases like pattern matching or is operator.

{brand: "MercedesBenz", plate: "DW 020 FN"} match {
  case is MyCar::Class -> true
  else -> false
}

Here we would expect true to be returned but it will return false as {brand: "MercedesBenz", plate: "DW 020 FN"} doesn't have the metadata class attached to it.

From here we saw 3 different approaches.

1. Having a new operator like (name to be defined) that only verifies for the type without the metadata.
2. Being able to specify optional metadata fields. For example instead of defining using {class?: "org.acme.MyClass"} so that now class is no longer required
3. Having a type prefix operator ^ that returns the type without the metadata

{brand: "MercedesBenz", plate: "DW 020 FN"} match {
  case is ^MyCar::Class -> true
  else -> false
}

In this case it will return true as ^MyCar::Class will match just the type properties without the class metadata

What do we think? I see more benefits on option 2 or 3. Are there other options?