Can we incrementally add new documents to index?

[kavhad]

With standard or baseline RAG it's simple to add new documents to the index as each document is unrelated, but since this is not the case for GraphRAG I assume a complete rebuild of the index is required or is their a feature for partial rebuild of the index based on what parts of the knowledge graph have changed?

[bmaltais]

Yes, you can. Just add the new document to the input folder and re-run the process. Only the new document will get entities created for it then it will be added to the summaries... That part will take some time unfortunatly... much more than I like ;-)

[mraguth]

But it supports only .txt files. Is it right?

[GaryDean]

From setting.yaml it seems as though you can specify the file pattern. So it should support files with .md extensions and other text formats.

input:
  type: file # or blob
  file_type: text # or csv
  base_dir: "input"
  file_encoding: utf-8
  file_pattern: ".*\\.txt$"

[TheBugKing]

the new index is created in a new folder with timestamp, when we have to query who would I know which index or artifact to load the data from for response

[kouskouss]

the new index is created in a new folder with timestamp, when we have to query who would I know which index or artifact to load the data from for response

I have a similar question. I performed two insertions of data into the same graphrag, which resulted in the creation of two different folders containing .parquet files. Where does the local search look for data? Which of the two folders should I use for the notebook examples involving local search?

[Rohith-Scalers]

if my new data is related to a particular community created in past will the new data added to the old community or will it create a new community . please suggest me any reference regarding this

[kShines]

@bmaltais

[natoverse]

We are working on a more robust incremental ingest approach, tracked in #741