Add BigQuery Loader V2

[pondzix]

No description provided.

[netlify]

✅ Deploy Preview for snowplow-docs ready!

Name	Link
🔨 Latest commit	dc1045e
🔍 Latest deploy log	https://app.netlify.com/sites/snowplow-docs/deploys/664c6ecadde50a0008164d4d
😎 Deploy Preview	https://deploy-preview-877--snowplow-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[stanch]

Could you also update this page? https://docs.snowplow.io/docs/storing-querying/storage-options/ BigQuery should be now an option for the 3 clouds

[stanch]

Suggested change

	:::tip
	You might find that some of your schemas were evolved incorrectly in the past, which results in the creation of these “recovery” columns after the upgrade. To address this for a given schema, create a new _minor_ schema version that reverts the breaking changes introduced in previous versions. (Or, if you want to keep the breaking change, create a new _major_ schema version.) You can set it to [supersede](/docs/understanding-tracking-design/versioning-your-data-structures/amending/index.md#marking-the-schema-as-superseded) the previous version(s), so that events are automatically validated against the new schema.
	:::

Do we also need instructions similar to these? https://docs.snowplow.io/docs/pipeline-components-and-applications/loaders-storage-targets/snowplow-rdb-loader/upgrade-guides/6-0-0-upgrade-guide/#identifying-schemas-that-need-patching @istreeter

[istreeter]

That section "Identifying schemas that need patching" was written long ago before we had the supercededBy feature. I would think now we have supercededBy, we shouldn't recommend to anyone to patch schemas.

Instead, the user can publish the fixed schema as a new major version, and configure the old schemas to automatically upgrade to the new major version. This is consistent with our belief that schemas are immutable. And it means we don't need to write a new igluctl feature.

[stanch]

My doubt was not specifically about patching but whether running igluctl would be useful to identify issues with schemas... Because I think it says what the incompatibilities are? With just a recovery column, how will the users know what to fix?

[istreeter]

igluctl has an existing sub-command verify-parquet

igluctl static verify-parquet <schema_dir>

It prints out warnings like

'com.test/test/jsonschema/1-0-1'. Changes: Incompatible type change String to Double at /id"

Unfortunately, it only works if the schemas are written into files on the local file system. It does not connect to remote Iglus.

We've not really planned what happens when existing users migrate to the new loader. And that is semi-deliberate: it is a lot of work to gather all the requirements and plan for all the things that could go wrong. So I've been delaying it until there is time to work on it properly.

[pondzix]

@stanch @istreeter as BQ Loader v2 relies on the same schema compatibility model/rules as parquet/Databricks/lake loader, we could use verify-parquet igluctl command and recommend it to users to detect incompatibilities. But this parquet in the name is confusing! Should we also do some renaming in igluctl then to make it more generic? And then also adjust docs

[stanch]

This will now need a rebase due to #915