Parse JSON array containing tuples

[hiromipaw]

I have to parse a JSON structure consisting of an array containing a tuple with a string and a Float64.

Ex:

{ "elapsed_seconds": [
              [
                "GENERAL:LAUNCHED",
                0.0
              ],
              [
                "GENERAL:EXTENDED",
                0.309999942779541
              ],
              [
                "GENERAL:EXTENDED",
                0.7200000286102295
              ],
              [
                "GENERAL:EXTENDED",
                0.9600000381469727
              ],
              [
                "GENERAL:BUILT",
                0.9600000381469727
              ],
              [
                "GENERAL:CLOSED",
                63.0
              ]
            ]
 }

I have been trying to use a list object but it seems that entails that all the elements of the list have the same data type.
Do you have any idea if it is possible to parse something like this? I have years of historical data, so I won't be able to easily change the JSON structure, even though I am aware these object are not the best possible way to write json.

[alamb]

I agree this will not be possible to represent as a ListArray in arryw

One possibility is to write your own JSON --> Arrow converter and hook it in as a TableProvider. While this might not be as easy/fast as the one in arrow-rs / DataFusion it would be very flexible.

You could map your JSON into either a Struct with fields like

Struct:
  'GENERAL:LAUNCHED': f64,
  'GENERAL:EXTENDED': f64,
...

Or maybe a map array for name/value pairs which is something like

ListArray {
  Struct {
   Name: String,
   Value: f64
  }
}

[hiromipaw]

Thanks! That could be a solution. Something I was wondering instead was why I cannot read a json field as stirng and decide to parse the content of that field at some later stage?

[hiromipaw]

I think I found the answer to my own question.

I can read the schema in arrow and coerce the primitive type to all elements of the list:

let mut json = arrow_json::ReaderBuilder::new(schema).with_coerce_primitive(true).build(BufReader::new(file)).unwrap();

And define the list with:

let elapsed_seconds = Field::new_list(
        "elapsed_seconds",
        Field::new_list(
            "event",
            Field::new("item", DataType::Utf8, true),
            true
        ),
        true
    );

So all items in the list are coverted to strings, which is good enough for what I need.