-
Notifications
You must be signed in to change notification settings - Fork 54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Part 4: read_table.c uses transform in ffi #614
base: main
Are you sure you want to change the base?
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #614 +/- ##
==========================================
- Coverage 83.45% 83.24% -0.21%
==========================================
Files 74 74
Lines 16877 17080 +203
Branches 16877 17080 +203
==========================================
+ Hits 14084 14218 +134
- Misses 2135 2201 +66
- Partials 658 661 +3 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the duckdb story with this new approach/PR? Have we explored that yet?
Asking because IIRC they push partition values down into their parquet reader, so they'll need to introspect the transforms and handle them differently than any kernel code we've written.
if (transformed_res.tag != OkHandleExclusiveEngineData) { | ||
print_error("Failed to transform read data.", (Error*)transformed_res.err); | ||
free_error((Error*)transformed_res.err); | ||
return NULL; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't we need to free data
and evaluator
regardless of whether this transform succeeded?
Stacked PR. Only review these commits
What changes are proposed in this pull request?
Use new transform functionality to transform data over FFI.
This lets us get rid of all the gross partition adding code in c :)
Still some things to work out here wrt. ffi types, so this is very much a draft.
But it works.
How was this change tested?