Releases: GoogleCloudPlatform/DataflowTemplates
Dataflow Templates 2024-06-04-00_RC01
Release week of 2024-06-03
Dataflow Templates 2024-06-04-00_RC00
Release week of 2024-06-03
Dataflow Templates 2024-05-28-00_RC00
Release Week of 2024-05-28
New Templates
N/A
Improvements
[datastream-to-spanner] Tests for additional CUJs (#1556)
[sourcedb-to-spanner] Add new feature (#1577, #1600, #1604)
[googlecloud-to-neo4j] Use built-in batch grouping transform (#1418)
[All Kafka] Add commit offsets (#1597)
[gcs-to-sourcedb] Add Integration test (#1562)
[kafka-to-kafka] Add authentication with SSL (#1555)
[spanner-change-streams-to-sharded-file-sink] Add Reverse replication reader IT (#1575)
Bug Fixes
[sourcedb-to-spanner] Fixed value mapping for blob and year (1578)
Contributors
Aditya Bharadwaj
Akarys Shorabek
Akshara Uke
Anand Inguva
Cloud Teleport
Deep1998
Dippatel98
djagaluru
Florent Biville
Jeffrey Kinard
jian jin
Manit Gupta
sharan-malyala
Sharan Teja M
Shreya Khajanchi
surabhiad
Svetak Sundhar
Tamanna Kakkar
Tanu Sharma
Vardhan Thigle
Vitaly Terentyev
Dataflow Templates 2024-05-21-00_RC00
New Templates
N/A
Improvements
- [BULK] Misc changes across types, logging and options
- [DataStreamToSpanner] - Convert sharded migration ITs to use Avro
- [Avro] - Add support for interval type
- [Plugins] - Add GcsReadBucket type
- [KafkaToBigQuery] - Add support for writing Kafka keys
- [Plugins] - Add support for KAFKA_TOPIC template parameters.
- Add Spanner PR workflow and exclude Spanner from global workflow
- [GcsToSourceDB] - Spanner Reverse replication writer IT
- [Plugins] - Template parameters: update old types to new ones
- [Documentation ] - Add more docs on adding ITs, flex templates, and deprecating parameters
- optimized the processing for lean Spanner write intervals
Bug Fixes
- [BULK MISC FIXES] - Add error counters; Fix null string handlings
Contributors
@ahmedabu98
@aksharauke
@akvelon
@an2x
@damccorm
@darshan-sj
@Deep1998
@Dippatel98
@manitgupta
@RajKumarGupta
@shreyakhajanchi
Dataflow Templates 2024-05-14-00_RC00
Release week of 2024-05-13
Dataflow Templates 2024-05-13-00_RC00
Release Week of 2024-05-13
New Templates
- Kafka to GCS template (#1448)
- Kafka To BigQuery Template (#1464)
- Python UDFs to MongoDB flex templates (#1497)
- Add Python UDF support to GCS to Splunk template (#1494)
Improvements
- [sourcedb-to-spanner] Unified Schema Mappings For Bulk Reader. (#1460)
- [jdbc] JDBC Table Schema Discovery (#1481)
- Upgrade beam version to 2.56.0 (#1504)
- [jdbc] Implementing JDBCIO.RowMapper to map ResultSet to SourceRow. (#1508)
- [bigtable-to-vectorembedding] Add custom App Profile support for Bigtable to Vector Embedding export (#1505)
- [spanner-migration] Add support for the unified custom logical types: JSON, NUMBER and VARCHAR for Spanner Migration Templates (#1509)
- Xlang Dockerfile speed improvements (#1499)
- [spanner-migration] Support UNION field shema types and improve debugging (#1524)
- [UI] Add parentName and parentTriggerValues attributes for templates (#1456)
Bug Fixes
- Parse secret name and password only when using Auth (#1512)
Contributors
Anand Inguva
Archana Gupta
Danny McCormick
Deep1998
Dippatel98
Jeffrey Kinard
Manit Gupta
Ron Gal
Sharan Teja M
Shreya Khajanchi
Supriya Koppa
Vardhan Thigle
Vardhan Vinay Thigle
Yi Hu
anikin
archanagupta03
fozzie15
gauravjainn
hugo-syn
sharan-malyala
surabhiad
vitaly.terentyev
Dataflow Templates 2024-05-08-00_RC00
Release week of 2024-05-06
Dataflow Templates 2024-04-23-00_RC00
Improvements
[YamlTemplate] speed improvements via distroless Xlang image
[JDBC] discover schema from information system tables
Bug Fixes
[Spanner] preserve null and empty arrays correctly
Contributors
Akshara Uke
Cloud Teleport
Damon
Damon Douglas
Danny McCormick
Deep1998
Jack McCluskey
Yi Hu
aksharauke
Dataflow Templates 2024-04-16-00_RC01
Release Week of 2024-04-16
Note: This release is being rolled out. It may not be in your region yet.
Improvements
- Beam version updated to Beam 2.55.1
- Support FLOAT32 type in v1 templates of Spanner
- YAML template performance improvements
Contributors
Aravind Pedapudi
Damon
Damon Douglas
Danny McCormick
Deep1998
Dippatel98
Dmitrii Burlutckii
Jeffrey Kinard
Manit Gupta
Vardhan Thigle
Yi Hu
djagaluru
fozzie15
sharantej
vitaly.terentyev
Dataflow Templates 2024-04-02-00_RC00
Release week of 2024-04-01