[TST] Regex testing

[Sicheng-Pan]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • Added a few unit tests for the literal expr impl
  • Added a basic proptest for ascii regex patterns
• New functionality
  • N/A

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[Sicheng-Pan]

• [ENH] Optimize regex algorithm #4624
• [ENH] Prefetch block by prefixes #4623
• [TST] Regex testing #4570 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[propel-code-bot]

Comprehensive Regex Unit Tests and Property-Based Testing for Literal Expressions

This PR introduces thorough unit and property-based (proptest) tests for the Rust regex literal expression implementation. Key updates include new test modules covering various literal matching scenarios, proptest strategies to generate random regex patterns and corresponding documents, and integration of these tests in blockfile metadata querying. Additional minor changes establish conversion traits for ChromaHir types and minor doc comment updates.

Key Changes:
• Added extensive #[cfg(test)] unit tests to rust/types/src/regex/literal_expr.rs, validating literal, class, and alternation matching logic
• Introduced proptest-based randomized regex generation for ASCII patterns and corresponding document data (see rust/types/src/strategies.rs)
• Implemented Arbitrary for ChromaHir and ChromaRegexTestDocuments, enabling recursive regex structure prop tests
• Added two proptest-based, property-driven regression tests in rust/segment/src/blockfile_metadata.rs, ensuring full-text index matches regex results
• Extended ChromaHir with From and From for regex_syntax::hir::Hir for smoother test integration and arbitrary pattern generation
• Minor doc improvements and corrections throughout regex modules

Affected Areas:
• rust/types/src/regex/literal_expr.rs (logic and tests)
• rust/types/src/regex/hir.rs (conversions, utility)
• rust/types/src/strategies.rs (proptest strategies, Arbitrary impls)
• rust/segment/src/blockfile_metadata.rs (integration and property tests)

Potential Impact:

Functionality: No changes to production code logic; only documentation and testing reliability are improved.

Performance: Negligible, as all additions are test-only. No measurable runtime or memory impact outside test runs.

Security: No security-relevant changes.

Scalability: Testing improvements may detect future scalability or matching regressions in regex literal implementations.

Review Focus:
• Examine property-based test coverage in strategies.rs for completeness and relevance.
• Assess unit test accuracy and logic in new tests in literal_expr.rs.
• Check the clarity and correctness of the Arbitrary derivations and conversions.
• Confirm the property-based tests in blockfile_metadata.rs create realistic synthetic data and properly compare against a reference regex engine.

Testing Needed

• Run cargo test --all to ensure new and old tests pass and proptests do not panic.
• Ensure proptests are not flaky under repeated runs or CI.
• Validate changes under full CI to ensure all affected modules build and test correctly.

Code Quality Assessment

rust/types/src/regex/literal_expr.rs: High; added tests are comprehensive and idiomatic.

rust/types/src/regex/hir.rs: High; From/Into implementations are clear and aid testing.

rust/types/src/strategies.rs: High; new test strategies well structured, uses proptest idioms correctly.

rust/segment/src/blockfile_metadata.rs: High; property-testing setup is clear and well integrated.

Best Practices

Testing:
• Property-based testing with proptest for robustness
• Unit tests cover realistic and edge-case scenarios
• Integration with CI via #[cfg(test)] and proptest! macro

Type-Safety:
• Uses explicit types and conversions, avoiding panics

Potential Issues

• Proptests, especially those generating recursive regexes, might occasionally generate expressions that are slow to match or process. Aborting or limiting test complexity is advisable.
• Default document generation may fail or generate invalid regex strings under rare situations; reviewer should ensure fallbacks handle empty or degenerate patterns.
• Performance during test runs may degrade with high document counts or complexity; ensure test parameters are balanced.

This summary was automatically generated by @propel-code-bot