Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Kernel] Remove the protocol specific conversions from JsonHandler.parseJSON #3613

Open
vkorukanti opened this issue Aug 27, 2024 · 1 comment
Labels
enhancement New feature or request good medium issue Good for those with Delta Lake experience kernel

Comments

@vkorukanti
Copy link
Collaborator

Feature request

Currently the JsonHandler.parseJson parses the stats values. Some of these values are for columns of type TIMESTAMP, DATE which are not native types in Json. Exactly how the string format of the TIMESTAMP, DATE can be interpreted depends on the Delta protocol details. We want to change it as follows:

  1. JsonHandler.parseJson just parses as native json types
  2. Kernel-API module applies expressions on the parsed JSON values to convert them to appropriate types.
@vkorukanti vkorukanti added enhancement New feature or request kernel good medium issue Good for those with Delta Lake experience labels Aug 27, 2024
@SemyonSinchenko
Copy link

Hello! I would be willing to contribute. Do I understand right, that changes should be done:

  • in io.delta.kernel.defaults.internal.data.DefaultJsonRow.decodeElement(JsonNode jsonValue, DataType dataType) I need to return String (jsonValue.textValue()) for "date" / "timestamp with tz" and Instant for "timestamp"
  • Applying expressions in this case should be moved to default implementation of ColumnarBatch? I see that JsonHandler.parseJson returns ColumnarBatch but I lost a little what happens after...

Thanks in advance for any advice/clue how to implement it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good medium issue Good for those with Delta Lake experience kernel
Projects
None yet
Development

No branches or pull requests

2 participants