Support Struct Conversion when reading Arrow data

[grundprinzip]

The following query fails:

_, err := spark.Sql(ctx, "SELECT struct(1, 'a') as col")
if err != nil {
    log.Fatalf("Failed: %s", err)
}

with the error:

2025/01/06 15:13:51 Failed: unsupported arrow data type STRUCT

[kronsbein]

Hi @grundprinzip, I’m new to Spark Connect Go, but want to start contributing. I’d like to take this up if it’s still available.

[grundprinzip]

Absolutely, please feel free to submit a PR!

[kronsbein]

Great, thank you, will do! I was wondering about how to implement this dynamically when the structs schema is unknown. Would it make sense to convert them into a map[string]any, using field names as keys and field values as values? Or would it be better to use reflection to create and populate a generic struct type dynamically at runtime?

[grundprinzip]

I think map[string]any is probably better because the receiver will have a hard time expecting the struct the right way.