Inquiry about jupyter notebook Spatial Reasoning with Point Clouds

[TikaToka]

Hello, thank you for sharing your fantastic work!

I am looking into the jupyter notebookSpatial Reasoning with Point Clouds, and I have a inquiry.

The Heatmap's quality is not good but makes sense, however the sampled coordinates makes no sense.

Do you know why this happens? Is this may be a discrepancy between original image and reshaped image?

---

Furthermore, I saw that depth image provided in jupyter is weird

how can it have only 4 types of value?

Thank you in advance

[TikaToka]

@remyxai no response?

[smellslikeml]

Hi @TikaToka, thanks for your support!
I apologize for the delayed response, updated the settings to receive notifications.

For the scenes we considered, the point sampling strategy works best when objects were rounder and not overlapped.

We're adding a few upgrades to this repo.
One change will prompt SAM2 using Florence-2 for bounding boxes grounded on captions instead of the points sampled from CLIPSeg's attention heatmap.

Unfortunately, it can still fail to separate overlapping objects like your test image

Regarding this depth image, it's encoding the metric depth inferred using ZoeDepth. To load it, I use open3D's RGBDImage.depth instead of Pillow