Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add RecursiveSplitter component for Document preprocessing #8605

Merged
merged 100 commits into from
Jan 10, 2025

Conversation

davidsbatista
Copy link
Contributor

Related Issues

Proposed Changes:

  • Adding a RecursiveSplitter, using a set of predefined separators to split text recursively - see issue for more details

How did you test it?

  • local unit tests and integration tests plus CI tests

Checklist

  • I have read the contributors guidelines and the code of conduct
  • I have updated the related issue with new insights and changes
  • I added unit tests and updated the docstrings
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I documented my code
  • I ran pre-commit hooks and fixed any issue

@github-actions github-actions bot added topic:tests type:documentation Improvements on the docs labels Dec 4, 2024
@coveralls
Copy link
Collaborator

coveralls commented Dec 4, 2024

Pull Request Test Coverage Report for Build 12712969950

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 1 unchanged line in 1 file lost coverage.
  • Overall coverage increased (+0.2%) to 91.26%

Files with Coverage Reduction New Missed Lines %
components/preprocessors/sentence_tokenizer.py 1 94.12%
Totals Coverage Status
Change from base Build 12710720579: 0.2%
Covered Lines: 8844
Relevant Lines: 9691

💛 - Coveralls

@davidsbatista davidsbatista marked this pull request as ready for review December 4, 2024 17:23
@davidsbatista davidsbatista requested review from a team as code owners December 4, 2024 17:23
@davidsbatista davidsbatista requested review from dfokina and julian-risch and removed request for a team December 4, 2024 17:23
@davidsbatista davidsbatista changed the title feat:: add recursive chunking strategy feat: add recursive chunking strategy Dec 4, 2024
@davidsbatista davidsbatista requested a review from sjrl December 4, 2024 17:24
@davidsbatista
Copy link
Contributor Author

@bglearning - mentioning you since I believe you were the one with most interest in this feature

@davidsbatista davidsbatista requested a review from sjrl January 10, 2025 16:03
@davidsbatista
Copy link
Contributor Author

@sjrl I did the final fixes, so that special characters are also counted as a word unit - I've also requested a new review from you

Copy link
Contributor

@sjrl sjrl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thanks for the work on this

@davidsbatista
Copy link
Contributor Author

@sjrl thanks! this was a long, complex on - let's keep an eye on this, I suspect some edge cases might still show up.

@davidsbatista davidsbatista merged commit 4f73b19 into main Jan 10, 2025
18 checks passed
@davidsbatista davidsbatista deleted the add-recursive-chunking branch January 10, 2025 16:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic:tests type:documentation Improvements on the docs type:feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a Recursive Chunking strategy
6 participants