Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify catalog reading #394

Merged
merged 4 commits into from
Oct 28, 2024
Merged

Simplify catalog reading #394

merged 4 commits into from
Oct 28, 2024

Conversation

delucchi-cmu
Copy link
Contributor

Closes #460

Puts all the file reading on loading a catalog into a single module. This reduces the complexity for reading the code, reduces code duplication, and also reduces the number of times we need to read the properties file, and perform file system stat calls to determine if files exist. This should speed up catalog loading on remote file systems where there is a latency associated with each file read.

Copy link

codecov bot commented Oct 25, 2024

Codecov Report

Attention: Patch coverage is 98.21429% with 1 line in your changes missing coverage. Please review.

Project coverage is 92.63%. Comparing base (f2704b8) to head (d5478fc).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/hats/loaders/read_hats.py 98.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #394      +/-   ##
==========================================
- Coverage   92.83%   92.63%   -0.20%     
==========================================
  Files          49       49              
  Lines        2134     2091      -43     
==========================================
- Hits         1981     1937      -44     
- Misses        153      154       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

github-actions bot commented Oct 25, 2024

Before [f2704b8] After [0b8a656] Ratio Benchmark (Parameter)
22.5±0.1ms 23.0±0.04ms 1.02 benchmarks.MetadataSuite.time_load_partition_info_order6
388±1ms 397±7ms 1.02 benchmarks.Suite.time_outer_pixel_alignment
93.5±1ms 94.7±0.6ms 1.01 benchmarks.MetadataSuite.time_load_partition_info_order7
126±0.5ms 127±0.4ms 1.01 benchmarks.time_test_alignment_even_sky
1.03±0.03ms 1.03±0.03ms 1.01 benchmarks.time_test_cone_filter_multiple_order
93.8±0.6ms 93.9±0.5ms 1 benchmarks.MetadataSuite.time_load_partition_join_info
110±0.9ms 109±1ms 0.99 benchmarks.Suite.time_paths_creation
46.5±0.5ms 46.3±0.6ms 0.99 benchmarks.Suite.time_pixel_tree_creation
15.1±0.8ms 13.9±0.9ms 0.92 benchmarks.Suite.time_inner_pixel_alignment

Click here to view all benchmarks.

Copy link
Contributor

@camposandro camposandro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Melissa, the loading logic has become much clearer already! Left just a couple of comments about outdated docstrings.

@delucchi-cmu delucchi-cmu merged commit 1710fd7 into main Oct 28, 2024
10 checks passed
@delucchi-cmu delucchi-cmu deleted the issue/460/read_hats branch October 28, 2024 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants