Introduce EndStreamAction in Delta Sharing Protocol #734
Conversation
linzhou-db
requested review from
littlegrasscao,
JialeTomTian,
chakankardb and
charlenelyu-db
PROTOCOL.md
Outdated
|
|
||
| ### readerFeatures | ||
| readerfeatures is only useful when `responseformat=delta`, it includes values from [delta reader | ||
| features](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#table-features). It's set by the | ||
| caller of `DeltaSharingClient` to indicate its ability to process delta readerFeatures. | ||
|
|
||
| ## API Response Format in Parquet | ||
| ### includeEndStreamAction | ||
| The key is `includeEndStreamAction` and the value is `true` of `false`, i.e. `includeEndStreamAction=true`. |
There was a problem hiding this comment.
NIT (spelling): true or false here?
PROTOCOL.md
Outdated
| Client sets `includeEndStreamAction=true` in the request header. | ||
|
|
||
| The server can: | ||
| 1) decide not to include `EndStreamAction` in the respnose, thus it has to set `includeEndStreamAction=false` or not set it in the response header. |
There was a problem hiding this comment.
Nit (spelling): response
| <tr> | ||
| <th>Client that doesn't specify the header</th> | ||
| <td colspan="2"> No changes in both request and response header, and the server will only return `EndStreamAction` at the | ||
| end of the response when needed*, the client shouldn't fail the request if not seeing the action in the response. </td> |
There was a problem hiding this comment.
Should we clarify what does needed mean here?
There was a problem hiding this comment.
added.
I intended to add the cases but forgot before sent out..
|
@charlenelyu-db @chakankardb please take a look when you got time, thanks! |
PROTOCOL.md
Outdated
| @@ -2512,7 +2512,7 @@ Accepted timestamp format by a delta sharing server: in the ISO8601 format, in t | |||
|
|
|||
| ## Delta Sharing Capabilities Header | |||
| This section explains the details of delta sharing capabilities header, which was introduced to help | |||
| delta sharing catch up with features in [delta protocol](https://github.com/delta-io/delta/blob/master/PROTOCOL.md). | |||
| delta sharing protocol evolve, such as with new feature support or catch up with delta features in [delta protocol](https://github.com/delta-io/delta/blob/master/PROTOCOL.md). | |||
There was a problem hiding this comment.
Let's chatgpt this:
This section explains the details of the Delta Sharing Capabilities header, which was introduced to enable the Delta Sharing protocol to evolve over time. This includes supporting new features and maintaining compatibility with advancements in the Delta protocol.
| - **delta**: format can be used to read a shared delta table with minReaderVersion > 1, which contains | ||
| readerFeatures such as Deletion Vector or Column Mapping. `delta-sharing-spark` libraries | ||
| that are able to process `responseformat=delta` will be released soon. | ||
|
|
There was a problem hiding this comment.
responseFormat
Specifies the expected format of the API Response in Parquet. Two values are supported:
parquet: Represents the response format used by delta-sharing-spark version 1.0 and earlier. This is the default format if responseFormat is not specified in the header. All existing Delta Sharing connectors are compatible with this format.
delta: Enables reading of shared Delta tables with minReaderVersion > 1, which may include advanced reader features such as Deletion Vectors or Column Mapping. Support for responseFormat=delta will be available in upcoming versions of the delta-sharing-spark library.
Introduce EndStreamAction and the header
includeEndStreamActionin Delta Sharing Protocol.