-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Simplify catalog reading #394
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #394 +/- ##
==========================================
- Coverage 92.83% 92.63% -0.20%
==========================================
Files 49 49
Lines 2134 2091 -43
==========================================
- Hits 1981 1937 -44
- Misses 153 154 +1 ☔ View full report in Codecov by Sentry. |
Click here to view all benchmarks. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Melissa, the loading logic has become much clearer already! Left just a couple of comments about outdated docstrings.
Closes #460
Puts all the file reading on loading a catalog into a single module. This reduces the complexity for reading the code, reduces code duplication, and also reduces the number of times we need to read the
properties
file, and perform file systemstat
calls to determine if files exist. This should speed up catalog loading on remote file systems where there is a latency associated with each file read.