chunking recommender

[mtawbiaws]

Issue #, if available:

Challenge

Chunking is the process of dividing documents into smaller sections, or "chunks," before embedding them into a Knowledge Base. This process enhances retrieval efficiency and precision. There are several chunking strategies available, each suited to different types of content and document structures. Examples of chunking strategies supported by Amazon Bedrock are:

• FIXED_SIZE: Splitting documents into chunks of the approximate size that set.
• HIERARCHICAL: Splitting documents into layers of chunks where the first layer contains large chunks, and the second layer contains smaller chunks derived from the first layer.
• SEMANTIC: Split documents into chunks based on groups of similar content derived with natural language processing.

FIXED_SIZE is useful in scenarios requiring predictable chunk sizes for processing. HIERARCHICAL chunking is appropriate when dealing with complex, nested data structures. Whereas Semantic Chunking is useful when dealing with complex, contextual information and processing documents where meaning across sentences is highly interconnected.
The main drawbacks of semantic chunking include higher computational requirements, limited effectiveness across different languages and scalability challenges with large datasets. The main drawbacks of hierarchical chunking include higher computational overhead, difficulty in managing deep hierarchies and slower query performance at deeper levels.

Selecting the right chunking strategy require understanding of benefits and limitations of each strategy in the context of analyzed documents, business requirements and SLAs. To determine the adequate chunking strategy, developer needs to manually assess document before selecting a strategy. The final choice is a balance between efficiency, accuracy, and practical constraints of the specific use case

Approach presented in this notebook

The approach presented in this notebook leverages Foundation Models (FMs) to automate document analysis and ingestion into an Amazon Bedrock Knowledge Base, replacing manual human assessment. The system automatically:

• Analyzes document structure and content
• Determines the optimal chunking strategy for each document
• Generates appropriate chunking configurations
• Executes the document ingestion process

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.