materialize-bigquery: JSON column type for array and objects #2324
Conversation
When the BigQuery materialization connector was initially built, BigQuery did not have a native JSON column type, so arrays and objects were materialized as "stringified" JSON. There is now a native JSON column type, so we can use that for these kinds of fields. The main complication here is preserving the behavior of pre-existing materializations, while allowing new ones to use the JSON column type. The `objects_and_arrays_as_json` feature flag was added previously, and that flag will toggle using the old stringified columns instead of the JSON columns. Most of the diff here is mechanical changes needed to thread through the value of that `objects_and_arrays_as_json` flag to where it needs to be.
williamhbaker
force-pushed
the
wb/mat-bq-json-col
branch
from
f0c7a35 to
ba0bfb1
Compare
| @@ -32,77 +35,86 @@ type transactor struct { | |||
| loggedStorageApiMessage bool | |||
| } | |||
|
|
|||
| func newTransactor( | |||
There was a problem hiding this comment.
Most of this diff is just from wrapping the newTransactor behavior in a closure so that it has access to the previously computed dialect, templates, and flag value.
There are no currently known cases where this should be allowed, and it is likely to break situations where the document is currently materialized as a string column and attempts to be migrated to a JSON column, where string -> JSON migrations results in a JSON string rather than an actual JSON object.
williamhbaker
force-pushed
the
wb/mat-bq-json-col
branch
from
2e75b1e to
be372c4
Compare
|
@williamhbaker regarding release process, why do we not set the flag on tasks now, and then release this with |
That's a good point, since I did this a little strangely and already added the flag in a prior PR we can do it like you suggest, and not have to have another follow-up. I'll update the PR and release process description for that! |
…fault New materializations created without the "no_objects_and_arrays_as_json" will now use JSON columns for objects and arrays.
|
@mdibaiee I finished updating this PR to have |
Description:
When the BigQuery materialization connector was initially built, BigQuery did not have a native JSON column type, so arrays and objects were materialized as "stringified" JSON.
There is now a native JSON column type, so we can use that for these kinds of fields.
The main complication here is preserving the behavior of pre-existing materializations, while allowing new ones to use the JSON column type. The
objects_and_arrays_as_jsonfeature flag was added previously, and that flag will toggle using the old stringified columns instead of the JSON columns.Most of the diff here is mechanical changes needed to thread through the value of that
objects_and_arrays_as_jsonflag to where it needs to be.The rollout of this change will be:
no_objects_and_arrays_as_json, which was added in materialize-bigquery: don't serialize JSON nulls #2323.Closes #2319
Closes #2333
Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)
Notes for reviewers:
Manually tested that an existing materialization can transition successfully from having no feature flag set to having the
no_objects_and_arrays_as_jsonset. Trying to setobjects_and_arrays_as_jsonon an existing materialization fails because of the newly-added compatibility check for existing root document columns, and the change to not allow them to be migrated. A brand new task withobjects_and_arrays_as_jsonset works fine with JSON columns from the start.This change is