Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix] High chunk_overlap causes last chunk to be entirely redundant #136

Merged
merged 2 commits into from
Jan 5, 2025

Conversation

bhavnicksm
Copy link
Collaborator

This pull request introduces changes to the src/chonkie/chunker/token.py file to improve the chunking process by refactoring how token groups are generated. The main change involves moving the _token_group_generator method to a more appropriate location within the file and integrating its usage more consistently across the codebase.

Key changes include:

  • Refactoring _token_group_generator:
    • Moved _token_group_generator method to a more appropriate location within the file.
    • Updated the chunk method to use _token_group_generator for calculating token groups.
    • Modified the _process_text_batch method to use _token_group_generator for generating token groups.

@bhavnicksm bhavnicksm merged commit 9389a4b into main Jan 5, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant