diff --git a/.semversioner/next-release/patch-20241031001404444046.json b/.semversioner/next-release/patch-20241031001404444046.json new file mode 100644 index 0000000000..10725e9652 --- /dev/null +++ b/.semversioner/next-release/patch-20241031001404444046.json @@ -0,0 +1,4 @@ +{ + "type": "patch", + "description": "add visualization guide to doc site" +} diff --git a/dictionary.txt b/dictionary.txt index 7ea41bd295..0288faac9a 100644 --- a/dictionary.txt +++ b/dictionary.txt @@ -96,6 +96,9 @@ onclick pymdownx linenums twemoji +Gephi +gephi +Gephi's # Verbs binarize @@ -183,4 +186,4 @@ kwds astrotechnician epitheg unspooled -unnavigated \ No newline at end of file +unnavigated diff --git a/docs/get_started.md b/docs/get_started.md index 46fec3feb4..c02273d50a 100644 --- a/docs/get_started.md +++ b/docs/get_started.md @@ -125,3 +125,6 @@ graphrag query \ ``` Please refer to [Query Engine](query/overview.md) docs for detailed information about how to leverage our Local and Global search mechanisms for extracting meaningful insights from data after the Indexer has wrapped up execution. + +# Visualizing the Graph +Check out our [visualization guide](visualization_guide.md) for a more interactive experience in debugging and exploring the knowledge graph. diff --git a/docs/img/viz_guide/gephi-appearance-pane.png b/docs/img/viz_guide/gephi-appearance-pane.png new file mode 100644 index 0000000000..0275b4f470 Binary files /dev/null and b/docs/img/viz_guide/gephi-appearance-pane.png differ diff --git a/docs/img/viz_guide/gephi-initial-graph-example.png b/docs/img/viz_guide/gephi-initial-graph-example.png new file mode 100644 index 0000000000..6e60c60b1e Binary files /dev/null and b/docs/img/viz_guide/gephi-initial-graph-example.png differ diff --git a/docs/img/viz_guide/gephi-layout-forceatlas2-pane.png b/docs/img/viz_guide/gephi-layout-forceatlas2-pane.png new file mode 100644 index 0000000000..23bdc5bb15 Binary files /dev/null and b/docs/img/viz_guide/gephi-layout-forceatlas2-pane.png differ diff --git a/docs/img/viz_guide/gephi-layout-pane.png b/docs/img/viz_guide/gephi-layout-pane.png new file mode 100644 index 0000000000..5dd58dcffd Binary files /dev/null and b/docs/img/viz_guide/gephi-layout-pane.png differ diff --git a/docs/img/viz_guide/gephi-network-overview-settings.png b/docs/img/viz_guide/gephi-network-overview-settings.png new file mode 100644 index 0000000000..03776de7d6 Binary files /dev/null and b/docs/img/viz_guide/gephi-network-overview-settings.png differ diff --git a/docs/visualization_guide.md b/docs/visualization_guide.md new file mode 100644 index 0000000000..3a249929de --- /dev/null +++ b/docs/visualization_guide.md @@ -0,0 +1,99 @@ +# Visualizing and Debugging Your Knowledge Graph + +The following step-by-step guide walks through the process to visualize a knowledge graph after it's been constructed by graphrag. Note that some of the settings recommended below are based on our own experience of what works well. Feel free to change and explore other settings for a better visualization experience! + +## 1. Run the Pipeline +Before building an index, please review your `settings.yaml` configuration file and ensure that graphml snapshots is enabled. +```yaml +snapshots: + graphml: true +``` +(Optional) To support other visualization tools and exploration, additional parameters can be enabled that provide access to vector embeddings. +```yaml +embed_graph: + enabled: true # will generate node2vec embeddings for nodes +umap: + enabled: true # will generate UMAP embeddings for nodes +``` +After running the indexing pipeline over your data, there will be an output folder (defined by the `storage.base_dir` setting). +- **Output Folder**: Contains artifacts from the LLM’s indexing pass. + +## 2. Locate the Knowledge Graph +In the output folder, look for a file named `merged_graph.graphml`. graphml is a standard [file format](http://graphml.graphdrawing.org) supported by many visualization tools. We recommend trying [Gephi](https://gephi.org). + +## 3. Open the Graph in Gephi +1. Install and open Gephi +2. Navigate to the `output` folder containing the various parquet files. +3. Import the `merged_graph.graphml` file into Gephi. This will result in a fairly plain view of the undirected graph nodes and edges. + +

+ A basic graph visualization by Gephi +

+ +## 4. Install the Leiden Algorithm Plugin +1. Go to `Tools` -> `Plugins`. +2. Search for "Leiden Algorithm". +3. Click `Install` and restart Gephi. + +## 5. Run Statistics +1. In the `Statistics` tab on the right, click `Run` for `Average Degree` and `Leiden Algorithm`. + +

+ A view of Gephi's network overview settings +

+ +2. For the Leiden Algorithm, adjust the settings: + - **Quality function**: Modularity + - **Resolution**: 1 + +## 6. Color the Graph by Clusters +1. Go to the `Appearance` pane in the upper left side of Gephi. + +

+ A view of Gephi's appearance pane +

+ +2. Select `Nodes`, then `Partition`, and click the color palette icon in the upper right. +3. Choose `Cluster` from the dropdown. +4. Click the `Palette...` hyperlink, then `Generate...`. +5. Uncheck `Limit number of colors`, click `Generate`, and then `Ok`. +6. Click `Apply` to color the graph. This will color the graph based on the partitions discovered by Leiden. + +## 7. Resize Nodes by Degree Centrality +1. In the `Appearance` pane in the upper left, select `Nodes` -> `Ranking` +2. Select the `Sizing` icon in the upper right. +2. Choose `Degree` and set: + - **Min**: 10 + - **Max**: 150 +3. Click `Apply`. + +## 8. Layout the Graph +1. In the `Layout` tab in the lower left, select `OpenORD`. + +

+ A view of Gephi's layout pane +

+ +2. Set `Liquid` and `Expansion` stages to 50, and everything else to 0. +3. Click `Run` and monitor the progress. + +## 9. Run ForceAtlas2 +1. Select `Force Atlas 2` in the layout options. + +

+ A view of Gephi's ForceAtlas2 layout pane +

+ +2. Adjust the settings: + - **Scaling**: 15 + - **Dissuade Hubs**: checked + - **LinLog mode**: uncheck + - **Prevent Overlap**: checked +3. Click `Run` and wait. +4. Press `Stop` when it looks like the graph nodes have settled and no longer change position significantly. + +## 10. Add Text Labels (Optional) +1. Turn on text labels in the appropriate section. +2. Configure and resize them as needed. + +Your final graph should now be visually organized and ready for analysis! diff --git a/mkdocs.yaml b/mkdocs.yaml index 5f30e7fa13..c2a04c081a 100644 --- a/mkdocs.yaml +++ b/mkdocs.yaml @@ -54,6 +54,7 @@ nav: - Microsoft Research Blog: "blog_posts.md" - Extras: - CLI: "cli.md" + - Visualization Guide: "visualization_guide.md" - Operation Dulce: - About: "data/operation_dulce/ABOUT.md" - Document: "data/operation_dulce/Operation Dulce v2 1 1.md"