-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Kernel] Add Default client implementations #1843
Conversation
de97c4e
to
0667019
Compare
kernel/kernel-default/src/test/java/io/delta/kernel/client/TestDefaultExpressionHandler.java
Outdated
Show resolved
Hide resolved
kernel/kernel-default/src/main/java/io/delta/kernel/client/DefaultExpressionHandler.java
Show resolved
Hide resolved
kernel/kernel-default/src/main/java/io/delta/kernel/client/DefaultFileReadContext.java
Outdated
Show resolved
Hide resolved
kernel/kernel-default/src/main/java/io/delta/kernel/client/DefaultFileSystemClient.java
Outdated
Show resolved
Hide resolved
kernel/kernel-default/src/main/java/io/delta/kernel/client/DefaultJsonHandler.java
Outdated
Show resolved
Hide resolved
|
||
private static Object decodeElement(JsonNode jsonValue, DataType dataType) | ||
{ | ||
if (jsonValue.isNull()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we check for nullable field here too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
its not needed here. We do a check before coming here. There are couple of places the check needs to be done when parsing the array element or parsing the map value element. Added a checks to throw exception of the null type is not expected.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought this is different from rootNode.get(name) == null
as it's a json literal null value?
kernel/kernel-default/src/main/java/io/delta/kernel/data/DefaultRowBasedColumnarBatch.java
Outdated
Show resolved
Hide resolved
kernel/kernel-default/src/main/java/io/delta/kernel/data/DefaultRowBasedColumnarBatch.java
Outdated
Show resolved
Hide resolved
kernel/kernel-default/src/test/java/io/delta/kernel/client/TestDefaultJsonHandler.java
Show resolved
Hide resolved
kernel/kernel-default/src/test/java/io/delta/kernel/client/TestDefaultJsonHandler.java
Show resolved
Hide resolved
kernel/kernel-default/src/main/java/io/delta/kernel/DefaultKernelUtils.java
Outdated
Show resolved
Hide resolved
kernel/kernel-default/src/main/java/io/delta/kernel/data/DefaultRowBasedColumnarBatch.java
Outdated
Show resolved
Hide resolved
49885c3
to
497a28b
Compare
Following client implementations for the default module are added: * `JsonHandler` * `ExpressionHandler` * `FileSystemClient` and the supporting classes.
497a28b
to
bc61406
Compare
currentFileReader = new BufferedReader( | ||
new InputStreamReader(stream, StandardCharsets.UTF_8)); | ||
} catch (Exception e){ | ||
Utils.closeCloseables(stream); // close it avoid leaking resources |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
propagate the exception?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Utils.closeCloseables
already throws an unchecked exception.
Related to this: The iterator interface needs to have some kind of checked exception throwing, but we can't override the interface. Added this to the list. We may need to revisit this. Unchecked exceptions are not always good.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't the original exception e
suppressed here though?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 on the iterator interface
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No its not.
if (exception == null) {
exception = new RuntimeException(ex);
}
else {
exception.addSuppressed(ex);
}
We keep the first exception (wrap it in RuntimeException because of the no-checked exceptions). Subsequent exceptions are added to suprrssed. they are still available, but won't be printed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I mean whatever exception caused us to reach the catch block here at all. Not sure if I'm missing something obvious
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we fail to open the file or something and stream = null
we just continue on here no?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, got it. Added a silent close version of the API. I think this exception e
is the main one we want to preserve.
kernel/kernel-default/src/test/java/io/delta/kernel/client/TestDefaultJsonHandler.java
Outdated
Show resolved
Hide resolved
kernel/kernel-default/src/main/java/io/delta/kernel/data/DefaultJsonRow.java
Outdated
Show resolved
Hide resolved
kernel/kernel-default/src/main/java/io/delta/kernel/client/DefaultJsonHandler.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Context
This PR is part of #1783.
Description
Following client implementations for the default module are added:
How was this patch tested?
UTs