Inserting a multi-part document results in duplicate nodes and edges

[jin38324]

I want to convert a long document into a Lightrage knowledge base.
If I convert the entire document directly, one error will lead to a total failure. For the purpose of risk control, the document is divided into multiple parts and insert is performed separately.
From the result data, it is found that there are a lot of duplications in nodes and edges.

After investigation, the reason is in the relevant code below.
LightRAG will merge nodes and edges with the same name when executing insert; but this is only valid for the same execution of _process_single_content. If you insert multiple documents like I did, the results will not be merged.

LightRAG/lightrag/operate.py

Line 391 in 874f3b3

_merge_nodes_then_upsert(k, v, knowledge_graph_inst, global_config)