Dataflow Templates 2022-08-29-00_RC00

Release Week of 2022-08-29

Improvements

[Spanner Change Stream] Allow setting autoscaling parameters
[Pub/Sub to Cloud Storage] Allow configuration of windowDuration parameter

Bug Fixes

[Spanner Change Stream to BigQuery] Fix parameter name from spannerRpcAuthority to rpcAuthority
[Spanner] Use Spanner version 6.23.3 to fix null pointer exceptions
[CDC] Better error handling when merge info can't be fetched from BigQuery
[BigQuery] Change - characters to _ in BigQuery dataset names

Contributors

@bvolpato
@marengaz
Internal contributors

Dataflow Templates 2022-08-15-00_RC02

Release Week of 2022-08-15

New Templates

N/A

Improvements

[ElasticSearchIO] ElasticsearchIO template improvements
[MongoDB] MongoDB batch template - Removed bucketing to avoid aggregation at read time
[Spanner] Created Spanner Resource Manager

Bug Fixes

[Spanner to BigQuery] Debug template options

Contributors

Mark Pevec
Venkatesh Shanbhag
ike-albert

Dataflow Templates 2022-08-08-00_RC01

Release Week of 2022-08-08

Improvements

[Dataplex Templates] Minor changes in the updateDataplexMetadata parameter description.

Bug Fixes

Fixed an issue with null error messages in BulkDecompressor (#433).

Contributors

@zhoufek
@an2x

Dataflow Templates 2022-08-01-00_RC04

Release Week of 2022-08-01 (second candidate)

Bug Fixes

[MongoDB] Fix classpath in spec files

Contributors

@an2x

Dataflow Templates 2022-08-01-00_RC02

Release Week of 2022-08-01

New Templates

• MongoDB to BigQuery

Improvements

• Expose numShards option in Pub/Sub to Text
• Updated static to parallel processing for MongoDB/BigQuery templates

Bug Fixes

• WindowedFilenamePolicy no longer changes the bucket name

Contributors

@an2x
@jrmccluskey
@pranavbhandari24
@theshanbhag

Dataflow Templates 2022-07-26-00_RC00

Release Week of July 25th, 2022

New Templates

[MongoDB] New MongoDB templates
[PubSubToText] New flex template that can handle both subscription and topic

Improvements

[DataplexFileFormatConversion] Support updating metadata in Dataplex for the newly generated data
[DataplexJdbcIngestion] Support updating metadata in Dataplex for the newly generated data.
[DatastreamToBigQuery] Bug-fix: Merge sort keys comparison should be wrapped by parentheses
[All] Upgrade Beam version to v2.39.0

Contributors

Venkatesh Shanbhag
@an2x
@oleg-semenov

Dataflow Templates 2022-07-18-00_RC00

Release Week of July 18, 2022

[Spanner templates] Improving Unit test coverage for spanner, spanner.ddl, spanner.common packages and AvroRecordConverter.
[Spanner templates] Moving generated classes from proto to a separate package.
[DatastreamToBigQuery] Use consistent BigQuery naming cleanup to avoid invalid characters.

Bug Fixes

[Pub/Sub to Splunk template] Disable failing test for HttpEventPublisher
[DatastreamToBigQuery] Add MergeInfo equals to avoid serialization warnings.

Contributors

@dhercher
@pranavbhandari24
djagaluru

Dataflow Templates 2022-07-12-00_RC00

Release Week of 2022-07-11

Bug Fixes

• [Dataplex Templates] Bugfix: set partition location to the generated file's parent directory instead of the full file path when updating Dataplex metadata.

Other

• Added jacoco plugin to the project for unit test coverage tracking.

Contributors

@an2x
@darshan-sj

Dataflow Templates 2022-07-04-00_RC00

Release Week of July 4, 2022

New Templates

[Spanner to BigQuery] Initial template creation for Spanner -> BigQuery

Improvements

[Spanner change streams to BigQuery] Respect the primary key ordinal position when doing stale read for UPDATE mod

Contributors

@ike-albert
@pabloem

Dataflow Templates 2022-06-27-00_RC00

Release Week of 2022-06-27

Improvements

[PubsubToText] Add inputSubscription parameter
[DataplexBigQueryToGcs] Support updating metadata in Dataplex for newly generated data.

Contributors

@andreigurau
@oleg-semenov
@an2x
@oleg-semenov
@pranavbhandari24