Support BigQuery JSON column type #5416
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5416 +/- ##
==========================================
- Coverage 61.24% 61.23% -0.01%
==========================================
Files 310 310
Lines 11058 11060 +2
Branches 751 736 -15
==========================================
+ Hits 6772 6773 +1
- Misses 4286 4287 +1 ☔ View full report in Codecov by Sentry. |
| @@ -213,6 +213,14 @@ private[types] object TypeProvider { | |||
| tq"_root_.java.lang.String @${typeOf[BigQueryTag]}", | |||
| q"{$rhs}.wkt" | |||
| ) | |||
| case q"$m val $n: _root_.com.spotify.scio.bigquery.types.Json = $rhs" => // Could not find how to mutualize | |||
There was a problem hiding this comment.
I ran some test. This branch is not matched (nor is the Geography one).
I'm wondering if toTable works in that case
There was a problem hiding this comment.
I only tested in real life fromTable (that I needed) TBH. But I may need toTable in the distant future.
There was a problem hiding this comment.
I'm trying to adapt the integration test to make sure those types work as expected
|
Thank you! |
|
It seems something does not work as expected with REPEATED JSON: I am getting two lines for the same entry, the JSON repeated data being splited between the two... |
|
I'm not sure I get the end state, can you give an example ? |
|
Nevermind, I did some tests and could not reproduce it. I'll open a bug if I find it back, it may not be related. |
Adds support for the JSON column type on BigQuery.
This mimics what has been done for
GEOGRAPHY: use a simplecase class Json(wkt: String)container. There are a couple of changes I have copied without knowing their use, so they may have none (like inStorageUtil).An alternative may be to store it into a Json parser implementation model (eg Jackson), however it would tie it to more complex things.