-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add verification tests #428
Conversation
Unit tests are failing because of #427. I need some guidance on the pre-commit failures. Most seem to be a result of path arguments being accepted as strings which are converted in the post_init, but the conversion isn't being recognized. |
b0b464a
to
1950217
Compare
1950217
to
7b2fb2f
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #428 +/- ##
==========================================
+ Coverage 94.57% 95.11% +0.54%
==========================================
Files 28 28
Lines 1622 1802 +180
==========================================
+ Hits 1534 1714 +180
Misses 88 88 ☔ View full report in Codecov by Sentry. |
@delucchi-cmu, this is finally ready for your review. I tested it on the ZTF DR20 Lightcurves dataset (29176 partitions; 10TB) several times. It requires about 60GB RAM with a bit over half of that coming from
It does not currently use Dask. If the times above are reasonably representative for other large catalogs (which they may or may not be; idk) then I don't think it needs it. Let me know what you think there. |
Co-authored-by: Melissa DeLucchi <[email protected]>
Change Description
Closes #118, closes #373, closes #374
Adds the following:
Verifier
classhats.io.validation.is_valid_catalog
, schemas, file sets, row countsStil to-do:
Code Quality
New Feature Checklist