Releases: pravega/flink-connectors
Pravega Flink Connectors 0.13.0
This release supports Flink 1.14, 1.15 and 1.16, and it is tested and validated with Pravega 0.13.0 release. The release artifacts of the supported Flink versions are available on Maven Central.
For more detail, please read the documentation on Pravega official website: http://pravega.io/docs/latest/connectors/flink-connector
Highlight features
- (#702) Support Flink 1.16
The connector fixes the compatibility issues and updates its API to adapt Flink 1.16
Enhancements
Bugfixes
- (#719) Explicitly cancel outstanding checkpoints when checkpoints get stuck
Flink 1.16 connector source code and artifacts are listed in the assets.
Here is the complete list of commits: changelog-flink.txt
Pravega Flink Connectors 0.12.0
This release supports Flink 1.13, 1.14 and 1.15, and it is tested and validated with Pravega 0.12.0 release. The release artifacts of the supported Flink versions are available on Maven Central.
For more detail, please read the documentation on Pravega official website: http://pravega.io/docs/latest/connectors/flink-connector
Highlight features
- (#651) Support Flink 1.15
The connector fixes the compatibility issues and updates its API to adapt Flink 1.15
Test improvements
- (#459) Migrate tests to testcontainer
- (#664) Migrate from JUnit4 to JUnit5
Flink 1.15 connector source code and artifacts are listed in the assets.
Here is the complete list of commits: changelog-flink.txt
Pravega Flink Connectors 0.11.0
This release supports Flink 1.12, 1.13 and 1.14, and it is tested and validated with Pravega 0.11.0 release
For more detail, please read the documentation on Pravega official website: http://pravega.io/docs/latest/connectors/flink-connector
Highlight features
- (#560) Support Flink 1.14
The connector fixes the compatibility issues and updates its API to adapt Flink 1.14
- (#460) Experimental: FLIP-143 new Sink
In the latest Flink 1.14 connector, we have implemented the new FLIP-143 Sink API to catch up with the Flink community latest updates. The features for the new API PravegaSink are the aligned with the legacy Sink API FlinkPravegaWriter including:
- Parallel writes
- Ordering guarantee based on routing key
- Data Durability
- Support transactional write
Note: This is only an experimental API, so there may still be changes in future versions. (e.g. Changing to FLIP-191 v2 Sink is on the plan(#653))
In the latest Flink 1.14 connector, we have implemented the new FLIP-27 Source API to catch up with the Flink community latest updates. The features for the new API PravegaSource are aligned with the legacy Source API FlinkPravegaReader including:
- Parallel reads
- Consistency
- Ordering guarantee based on routing key
- Seamless checkpointing integration with Pravega checkpoint
Note: This is only an experimental API, so there may still be changes in future versions.
Notable bugfixes
- (#635) Fix null pointer when closing pravega batch client factory
- (#639 #642 #647) Refinements for document
Notifications
- (#675) Update schema registry to 0.4.0
Along with this fix, withDeserializationSchemaFromRegistry API is now dropped and now the actual produced type should be provided when deserialization schema is initialized.
- (#617) Increase default transaction lease
This fix increased the default transaction lease for transactional sink from 2 to 10 minutes.
Flink 1.14 connector source code and artifacts are listed in the assets.
Here is the complete list of commits: changelog-flink.txt
Pravega Flink Connectors 0.10.1
This release contains exciting new features and significant enhancements over the previous major release line (0.9.x). This page provides an overview of major ones.
This release supports Flink 1.11, 1.12 and 1.13, and it is tested and validated with Pravega 0.10.1 release
For more detail, please read the documentation on Pravega official website: http://pravega.io/docs/latest/connectors/flink-connector
Highlight features
The connector fixes the compatibility issues and updates its API to adapt Flink 1.12 and Flink 1.13.
- (#424) Pravega Catalog support
In the latest Flink 1.12 and 1.13 connector, Catalog API, also known as FLIP-30, is implemented in the connector with the help of Pravega and its Schema Registry. Schema registry provides Pravega stream metadata such as schema and serialization that the connector can accommodate and then present in Flink as a typical database catalog. Once the schemas are registered and the Flink catalog is set, Pravega streams are automatically mapped into tables without the CREATE TABLE SQL DDL.
To create such a catalog, just have the schema registry set up and try with CREATE CATALOG DDL statement with the options listed here.
- (#530) Table API Debezium support
Since Debezium 1.6.0 release, Pravega has been one of the supported sink for Debezium server, and Flink can act as a query engine to perform continuous queries with the ever-changing table. After the Flink 1.13 connector implements the multiple rows emission in deserialization, users can now use the official Flink debezium-json format to consume the CDC data in Pravega streams.
- (#494) FLIP-107: Expose metadata for new Table source
With FLIP-107, Flink has added the support for metadata along with the event for both table source and sink. Pravega Flink connector integrates the API to export metadata from both the connector and format. Furthermore, the EventPointer in bytes is provided as metadata in the table source.
- (#571) Add Python datastream API support
Python DataStream API connector is now integrated as an experimental feature. The basic idea is to use py4j to call the Java implementation. The suggested implementation for such a python wrapper is offered in the code base. Users can include the python files in the project and follow the guide to develop the pyflink application.
- (#445) Adopt github actions for CI pipeline
We have transitioned our CI pipeline from Travis to Github Actions to have it more stable.
Notable bugfixes
- (#441) Fail to reset reader group with no checkpoint recovery
- (#517) Fix NPE issue for recovery job after enabling transactions
- (#481) Row-based router cannot be specified on non-null field
Notifications
- (#556) Drop support for Scala 2.11
Flink community is trying to drop support for Scala 2.11 for some reasons (Detail in: https://issues.apache.org/jira/browse/FLINK-20845) and we also receive some feedback that Scala 2.11 is less used. We have decided to no longer release the Scala 2.11 artifacts for the connector. For users still on Scala 2.11, it's now required to build the connector on your own to use.
- (#493) Make reader group optional in table API
If the reader group name is set in Flink table API, the table cannot be reused as duplicate createReaderGroup calls would mess up. It is suggested not to use this scan.reader-group.name option and the connector can give a random one so the Flink table can be reused.
- (#490) Use maven publish instead of maven plugin in gradle
The Gradle maven plugin is deprecated according to the offical website and is recommended to use the new maven-publish plugin. Developers need to pay attention to the packaging script changes accordingly.
Flink 1.13 connector source code and artifacts are listed in the assets.
Here is the complete list of commits: changelog-flink.txt
Pravega Flink Connectors 0.9.1
0.9.1 release is a bug-fix release since the 0.9.0 release
Notable bugfixes:
This release supports Flink 1.9, 1.10 and 1.11, and it is tested and validated with Pravega 0.9.1 release
Flink 1.11 with Scala 2.12 connector source code and artifacts are listed in the assets. The release artifacts of all the supported Flink and Scala versions are available on Maven Central.
Here is the complete list of commits: changelog-flink.txt
Pravega Flink Connectors 0.9.0
This release contains exciting new features and significant enhancements over the previous major release line (0.8.x). This page provides an overview of major ones.
This release supports Flink 1.9, 1.10 and 1.11, and it is tested and validated with Pravega 0.9.0 release
For more detail, please read the documentation on Pravega official website: http://pravega.io/docs/latest/connectors/flink-connector
Highlight features
- (#392) FLIP-95/FLIP-122 new Flink Table API in Flink 1.11
In the latest Flink 1.11 connector, new Table API described in FLIP-95 is supported, a simpler CREATE TABLE SQL DDL can be used to create the table-stream connection instead of the descriptor style for legacy Table API. e.g.
CREATE TABLE test (
user_id STRING,
item_id BIGINT)
with (
'connector' = 'pravega'
'controller-uri' = 'tcp://localhost:9090',
'scope' = 'scope',
'scan.execution.type' = 'streaming',
'scan.reader-group.name' = 'group1',
'scan.streams' = 'stream',
'sink.stream' = 'stream',
'sink.routing-key.field.name' = 'user_id',
'format' = 'json'
)
Introduction with more details can be found in the official document.
- (#422) Add schema registry support for streaming and batch reader
The connector can integrate with Pravega schema registry, and users can use below method to use the schema registry deserializer to read events from streams
// e.g. For avro generic record in streaming source,
FlinkPravegaReader.<GenericRecord>builder()
....
.withDeserializationSchemaFromRegistry(streamName, GenericRecord.class)
.build()
- (#426) Upgrade to Java 11
The compatibility for Java 8 is still kept, so it is still fine to use the artifact under Java 8 environment. The main difference is when developing this project and running the unit tests, Java 11 is required.
Notable bugfixes
- (#377) Refactor Pravega Sink to extend TwoPhaseCommitSinkFunction
- (#433) Make event router optional
- (#455) Increase the default transaction lease renewal time to 120 seconds
Notifications
- (#438) Legacy table API is deprecated, please upgrade to Flink 1.11 connector and use the new Table API.
- (#439) Deprecated
PravegaSerializationis removed, please use the serialization/deserialization schema instead.
Flink 1.11 with Scala 2.12 connector source code and artifacts are listed in the assets.
Here is the complete list of commits: changelog-flink.log
Pravega Flink Connectors 0.8.1
Release 0.8.1 of Flink Pravega Connectors
0.8.1 is a bug-fix release since the 0.8.0 release
Notable bugfixes:
- (#416) Zombie GRPC threads in Flink job cancel
This release supports Flink 1.8, 1.9 and 1.10, and it is tested and validated with Pravega 0.8.1 release
For more detail, please read the documentation on Pravega official website: http://pravega.io/docs/latest/connectors/flink-connector
Flink 1.10 with Scala 2.12 connector source code and artifacts are listed in the assets.
Here is the complete list of commits: changelog-flink.txt
Pravega Flink Connectors 0.8.0
Release 0.8.0 of Flink Pravega Connectors
Highlight features:
Notable bugfixes:
- (#409) Fix Dynamic-loading credentials for Pravega client in Table API
- (#399) Better error handling for RuntimeException in streaming source
- (#405) Resource cleanup to avoid thread leak for Pravega Client Factory
Notifications:
- (#347) Desciptor API is moved into
table.descriptorspackage - (#215) Deprecated Json format table API is removed, please use the descriptor API instead.
This release supports Flink 1.8, 1.9 and 1.10, and it is tested and validated with Pravega 0.8.0 release
For more detail, please read the documentation on Pravega official website: http://pravega.io/docs/latest/connectors/flink-connector
Flink 1.10 with Scala 2.12 connector source code and artifacts are listed in the assets.
Here is the complete list of commits: changelog-flink.txt
Pravega Flink Connectors 0.7.1
Release 0.7.1 of Flink Pravega Connectors
Highlight features:
Notable bugfixes:
- (#364) Makes
writer.enable-watermarkoptional in Table API
Notifications:
- (#347) Desciptor API is moved into
table.descriptorspackage - (#215) Deprecated Json format table API is removed, please use the descriptor API instead.
This release supports Flink 1.8, 1.9 and 1.10, and it is tested and validated with Pravega 0.7.1 release
For more detail, please read the documentation on Pravega official website: http://pravega.io/docs/latest/connectors/flink-connector
Here is the complete list of commits: changelog-flink.txt
Pravega Flink Connectors 0.7.0
Release 0.7.0 of Flink Pravega Connectors
Highlight features of this release:
- Supports Flink major versions 1.7, 1.8 and 1.9
- (#180) Supports deserialization with metadata from Pravega
Notable bugfixes:
- (#319) Fixes
TruncatedDataExceptionhandling - (#331) Fixes infrequent recover failure from savepoints
- (#343) Treats watermark in an inclusive manner
Notifications:
- (#179)
PravegaSerializationis deprecated and will be removed in future releases
This release is tested and validated with Pravega 0.7.0 release
Here is the complete list of commits
flink-connector-0.7.0-release-notes.txt