-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adds fine filtering on read_hipscat
#350
Conversation
Click here to view all benchmarks. |
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #350 +/- ##
==========================================
+ Coverage 98.32% 98.33% +0.01%
==========================================
Files 43 43
Lines 1430 1440 +10
==========================================
+ Hits 1406 1416 +10
Misses 24 24 ☔ View full report in Codecov by Sentry. |
Adds fine filtering to the
read_hipscat
operation. Previously, when provided with asearch_filter
, the catalog was loaded with the pixels within the specified spatial region but we did not filter their data. This behavior created confusion with the users so we decided to apply the full search. We still do it in two steps, first figuring out which subset of files we should read from disk, creating the Dask DataFrame with the partitions, and then performing filtering on each of them. Closes #308.