Custom chunking strategy to be used with graphrag

[vchadha007]

Hi, I want to chunk my own text and then use graphrag on the text units I produce using a custom strategy. Is that something doable?

[l4b4r4b4b4]

1. Chunk your documents with some external library
2. Put them in a csv in input

This should work no? 🤔

[vchadha007]

For a small POC, sure. You could chunk up the docs using an external lib and place each chunk doc individually in your input folder, but what about thousands of documents? I think we should have a wrapper around the langchain splitters so that we can just place the text documents in their raw format and choose out of multiple chunking techniques.

[l4b4r4b4b4]

Would rather use unstructured

[AxithChoudhary]

yeah you can do that i have done it for that you need to change token.py file there you will get the chunk code
\Lib\site-packages\graphrag\index\verbs\text\chunk\strategies\tokens.py

this is my paragraph level chunking you can change with yours

Paragraph chunking

def split_text_on_tokens(
texts: list[str], enc: Tokenizer, tick: ProgressTicker
) -> list[TextChunk]:
"""Split incoming text and return chunks based on sentence boundaries."""
log.info("we are in token.py")
result = []
chunked_texts = []

# Split each text by sentence boundary regex
for source_doc_idx, text in enumerate(texts):
    # Split text on sentence boundaries, preserving sentence boundaries
    sentences = re.split(r'(?<=\.)\s*\n', text)
    chunked_texts.extend([(source_doc_idx, sentence) for sentence in sentences])
    tick(1)

# Encode each chunk and create TextChunk objects
for source_doc_idx, chunk_text in chunked_texts:
    encoded = enc.encode(chunk_text)
    chunk_ids = [(source_doc_idx, id) for id in encoded]
    chunk_text = enc.decode([id for _, id in chunk_ids])
    result.append(
        TextChunk(
            text_chunk=chunk_text,
            source_doc_indices=[source_doc_idx],
            n_tokens=len(chunk_ids),
        )
    )

log.info(f"{len(result)} chunks created")
log.info(f"{result}")

return result