Unsupported json/dynamic serialization

[kurayama]

What happened:
I tried changing the opentelemetry-collector-contrib exporter to use JSON instead of Map(String,String) for LogAttributes and ResourceAttributes.
Outputting some JSON values throws an error.

What you expected to happen:
All JSON values correctly showing.

How to reproduce it (as minimally and precisely as possible):
Change the otel schema to use JSON instead of Map(String,String) for LogAttributes
Selecting the root JSON value breaks in my scenario, some other nested JSONs also break.

Screenshots

Environment:

• Grafana version: 11.5
• Plugin version: 4.8
• OS Grafana is installed on: docker
• User OS & Browser: macOS and Safari

[SpencerTorres]

I actually just made a fix for this today in ClickHouse/clickhouse-go#1501 and ClickHouse/ch-go#1043

We should be able to get the fix propagated/released in the next few days. Thanks!

As a short term fix you can try casting to Array(String) perhaps?

[adamyeats]

Closing as #1172 was merged and released to resolve this.

[kurayama]

@adamyeats @SpencerTorres still have the same error on the latest release
isn't the fix in ch-go v0.65.1 instead of v0.65.0?

[SpencerTorres]

@kurayama The JSON type is implemented separately in clickhouse-go, ch-go only supports JSON strings. The correct version is included in plugin version 4.8.1

I have personally tested a query that works with Array(JSON), specifically from an OTel table.

Can you confirm again which version you're using for the plugin?

[kurayama]

Unless I'm doing something wrong, seems to have the same error 🤔
It's a brand new grafana docker container running the latest version.
It works as expected when the same query is done via clickhouse client.

[SpencerTorres]

Fascinating. Yeah that's broke

Can you run this:

SELECT arrayJoin(distinctJSONPathsAndTypes(LogAttributes)) FROM otel_logs_new

Curious to see what the paths and types are. If the paths are secret you can just do a.b.c, just want to see the nesting and all that. Thanks!

[SpencerTorres]

I have a reproducible query here:

SELECT ['{"x":5}','{"y":6}']::Array(JSON)::Dynamic as array_json

Looking into it now. This is specifically with Array(JSON) inside Dynamic 👀

[SpencerTorres]

Fixed in ClickHouse/clickhouse-go#1506

Going to release a patch ASAP, then we can update the plugin again.

The original fix supported Array(JSON), this one supports your specific column being Array(JSON)::Dynamic, since all types inside a JSON are Dynamic.

Appreciate your patience, these types can be tricky to decode. I had to get the hex editor out to see what was going on

[kurayama]

Thanks for looking into it.
I'll test again once the patch comes out.

[SpencerTorres]

Released the clickhouse-go patch in https://github.com/ClickHouse/clickhouse-go/releases/tag/v2.32.2

Grafana team should be able to get a release soon. Thanks!

[SpencerTorres]

@kurayama v4.8.2 has been released, let me know if this solves your issues. Thanks!

[kurayama]

@SpencerTorres I confirm it correctly works now 👌