Drop NULL columns from INSERT INTO

[ryannedolan]

Summary

Drop NULL columns (e.g. NULL AS KEY) from INSERT INTO pipelines.

Promote NULL fields to BYTES in DDL (e.g. KEY BYTES).

Details

Flink does not allow NULLs in queries (SQL) or in table definitions (DDL). This is problematic, because we would like to be able to write NULL AS KEY in subscriptions.

In Hoptimator, sink tables are derived from subscription SQL. If we write NULL AS KEY in a subscription, the sink table would have a NULL-typed column, and the Flink job would fail. Instead, NULL AS KEY should result in the column being omitted form the pipeline entirely (defaulting to NULL).

At the same time, Flink DDL cannot have NULL-typed fields, so we promote these to BYTES.

The result is that SELECT ... NULL AS KEY ... results in a sink table with KEY BYTES and pipelines that omit the KEY field (defaulting to NULL).

Testing

New unit tests. Tested locally and by manually tweaking production pipelines.

Changed integration tests to use NULL AS KEY.