Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parquet test improvements #175

Merged
merged 10 commits into from
Oct 13, 2023
Merged

Parquet test improvements #175

merged 10 commits into from
Oct 13, 2023

Conversation

stanbrub
Copy link
Collaborator

@stanbrub stanbrub commented Oct 3, 2023 •
edited
Loading

Rewrote the existing Parquet tests.

  • Added parquet write tests for single-column types string, BigDecimal, long, int, short, array, vector
  • Added read/write tests for multiple columns
  • Wrote a specialized test runner for reusing column generation definitions

Note: The ParquetTestRunner departs from the StandardTestRunner in that it generates data directly in Deephaven engine. So the data isn't reused by any other test set. This was necessary to allow "read" tests to use test data generated by "write" tests and to be able to test Deephaven-specific vectors that are not supported through AVRO.

@stanbrub stanbrub linked an issue Oct 3, 2023 that may be closed by this pull request
Parquet Benchmark Test Scale Improvements #115
Closed
@stanbrub stanbrub self-assigned this Oct 3, 2023
@stanbrub stanbrub merged commit 20a79ec into deephaven:main Oct 13, 2023
2 checks passed
@stanbrub stanbrub deleted the parquet-test-improvements branch October 13, 2023 21:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@malhotrashivam malhotrashivam malhotrashivam approved these changes

Assignees

@stanbrub stanbrub

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Parquet Benchmark Test Scale Improvements
2 participants
@stanbrub @malhotrashivam