Create topological graph for image segments

GraphVision

A library facilitating the generation of topological representations for image segments. The topological graph is created based on segments and their visual embeddings as nodes. This segment topology retains not only spatial but also semantic information, making it very useful for various tasks in vision, localization, robotics[1] and Game Development.

GraphVision provides graphical representations that enable us to perform visual queries and interact with our segmentation graph without the usual preprocessing hassle. GraphVision handles everything and also offers functionalities to visualize the segment topology and perform visual queries on the graph.

Usage

Go though the Colab Notebook to get a detailed overview of the library.

The library requirements are flexible but still its suggested to use a virtual environment.

pip install graphvision

Read an Image

image_bgr = cv2.imread("image.jpg")
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)

Initialize the generator, call the segmentation model, generate the masks and that is it! You are ready create the topological graph for your segments.

from graphvision import Generator, SegmentGraph
gen = Generator()
mask_gen = gen.SAM()
segments = gen.generate_masks(mask_gen, image_rgb)

You can visualize or plot it using gen.plot_segments(segments)

Create the topological graph

sg = SegmentGraph(segments, image_rgb)
G, centroids = sg.get_topology(dist_thres=150) # Other Optional Parameters: area_percent, add_to_bbox

The graph G can be returned as a networkx (default) or as a PyTorch geometric object. This graphical representation can now be leveraged in various vision tasks such as object localization and environment mapping in robotics, based on both spatial and semantic features..

Query the segment graph

You can also perform visual queries on the graph to locate objects dependent on other nodes (neighbouring objects). This a naive implementation, so please go through the Colab notebook to understand it in greater detail.

result = sg.query_segment_graph(G, query_pair=("Girl", "plant"), show_legend=True)

Reference

1. RoboHop: Segment-based Topological Map Representation for Open-World Visual Navigation