TriangleMeshReceptacles and RearrangeGenerator Improvements #1108

aclegg3 · 2023-02-01T00:35:19Z

Motivation and Context

New features include:

TriangleMeshReceptacle class - area weighted uniform sampling of 3D positions from an arbitrary triangle mesh surface. Can be loaded from .PLY format.
Sampling efficiency improvement: Added option to remove unstable objects instead of throwing away the episode if minimum number of objects for each sample can still be satisfied.
Improved reporting of sampling statistics and timings for RearrangeGenerator.
Added debug images of unstable objects to improve receptacle/scene debugging.
triangle sampling and "point in triangle" test added to geometry_utils and tested

Changes include:

Receptacle debugging visualization now uses DebugLineRender instead of attaching wire-frames to objects.
bugfix: max number of sample attempts were not reset after successful pairing.
bugfix: save file paths for DebugVisualizer were not always propagated correctly.
new and improved docstrings and typing

How Has This Been Tested

Added unit tests specifically for parsing and sampling from AABBReceptacles and TriangleMeshReceptacles.
Must be tested against habitat-sim PR 1994 with modified test assets.

Types of changes

Docs change / refactoring / dependency upgrade
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

Checklist

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have completed my CLA (see CONTRIBUTING)
I have added tests to cover my changes.
All new and existing tests passed.

…nd improved debugging

…sets

aclegg3 · 2023-02-01T00:37:26Z

.circleci/config.yml

+              git fetch --all
+              git checkout receptacle-test-assets


Testing only: this will be removed before merge.

…ments

…dd tests

…ments

habitat-lab/habitat/datasets/rearrange/samplers/receptacle.py

mosra · 2023-02-10T19:00:53Z

habitat-lab/habitat/datasets/rearrange/samplers/receptacle.py

+    ), "Must be a triangle mesh."
+
+    # zero-copy reference to importer datastructures
+    mesh_data: Tuple[List[mn.Vector3], List[int]] = (


If you'd want to be exact with the typing annotation, the actual type is cr.containers.StridedArrayView1D, not a List, but right now I don't have the [] "templating" implemented so saying cr.StridedArrayView1D[mn.Vector3] would probably cause Python to complain.... and without it it's maybe too nondescriptive (like np.ndarray is). I'll investigate what can be done on that front (implementing __getitem__ on the typeclass? huh).

mosra · 2023-02-10T19:11:04Z

habitat-lab/habitat/datasets/rearrange/samplers/receptacle.py

+                dblr.draw_transformed_line(
+                    verts[edge], verts[(edge + 1) % 3], color
+                )
+        dblr.pop_transform()


Just FYI, the dedicated tool for drawing wireframes (without having to manually compose them out of line segments) is the MeshVisualizer shader. I think it was used somewhere in the C++ viewer already, but no idea how easy is it to make it available here.

In case the shader would be available, you'd keep the original MeshData (instead of taking just views on positions and indices from it) and draw it directly with the shader.

Cool suggestion, I think this would be a cool way to do some debugging visualization in the future. Construct MeshData for the debug mesh and then register it for the MeshVisualizer shader.
Is this exposed to python in some way that it could draw on top of an existing framebuffer or would it need to be built into the Habitat rendering pipeline on the C++ side?

The shader isn't exposed at the moment (but I could do that easily), and yes that would work as long as you either have the framebuffer currently bound or have a way to access & bind it for drawing.

On the other hand, such direct approach would be very incompatible with the batch renderer ... so for easier porting in the future it's probably best to just leave it as-is 😅 The debug line drawing will have to get ported to the batch renderer in any case, and I could then maybe expand it to handle also wireframe meshes somehow.

…ments

vincentpierre

Looks great. Left some comments, but I do not have much to say about the overall change.

habitat-lab/habitat/datasets/rearrange/rearrange_generator.py

habitat-lab/habitat/datasets/rearrange/samplers/receptacle.py

vincentpierre · 2023-02-27T19:20:14Z

habitat-lab/habitat/datasets/rearrange/samplers/receptacle.py

+    # NOTE: We don't support mesh merging or multi-mesh parsing currently
+    if importer.mesh_count > 1:
+        raise NotImplementedError("TODO: multi-mesh receptacle support.")


Suggested change

# NOTE: We don't support mesh merging or multi-mesh parsing currently

if importer.mesh_count > 1:

raise NotImplementedError("TODO: multi-mesh receptacle support.")

# TODO: We don't support mesh merging or multi-mesh parsing currently

if importer.mesh_count > 1:

raise NotImplementedError("Importing multi-mesh receptacles (mesh merging or multi-mesh parsing) is not supported. ")

I could expose MeshTools::concatenate() and SceneTools::flattenMeshHierarchy() for doing this if you want.

I think folks are hoping to eventually have multiple separate meshes in a single file which we'll split out here.
Generally happy to see more Magnum mesh tools in python though. 🙂

Update: I think we want to store separate receptacle meshes as glTF nodes. So here we'll load the glTF file and iterate over all meshes in the scene, creating a TriangleMeshReceptacle for each one.

habitat-lab/habitat/utils/geometry_utils.py

…entered at COM

aclegg3 · 2023-03-11T19:11:44Z

Looks great. Left some comments, but I do not have much to say about the overall change.

Thanks, I've addressed your comments and resolved those not containing additional discussion.

vincentpierre

LGTM, I would prefer another approval from @mosra if possible.

vincentpierre · 2023-03-13T17:52:47Z

habitat-lab/habitat/datasets/rearrange/configs/all_receptacles_test.yaml

@@ -0,0 +1,54 @@
+---


Where is this file used? I could not found a test that was using it.

This is for users to test generation on their scenes with all receptacles defined without needing to define another custom generator config. I can remove it from this PR if that isn't a good enough reason to include it.

…tage.

…ments

…research#1108) * updates to rearrange episode generation to support mesh receptacles and improved debugging * docstrings and typing for DebugVisualizer util * mesh receptacle debug_draw with global transforms * add some tests for sampling accuracy * bugfix - use default_sensor_uid instead of hardcoded "rgb" in DebugVisualizer * adjust debug peek target to bb center to better capture objects not centered at COM * add debug circles to the peek API * add peek_scene variant for easliy getting images of a full scene or stage.

* TriangleMeshReceptacles and RearrangeGenerator Improvements (#1108) * updates to rearrange episode generation to support mesh receptacles and improved debugging * docstrings and typing for DebugVisualizer util * mesh receptacle debug_draw with global transforms * add some tests for sampling accuracy * bugfix - use default_sensor_uid instead of hardcoded "rgb" in DebugVisualizer * adjust debug peek target to bb center to better capture objects not centered at COM * add debug circles to the peek API * add peek_scene variant for easliy getting images of a full scene or stage. * RearrangeGenerator cleanup and unstable object culling (#1248) * fix unstable object culling compatibility with target sampling and change default to use this feature * add comments and typing * fix bug where only the first of a target sampler's object samplers was used * Add support for glTF multi-receptacle assets (#1249) * add support for glTF multi-receptacle assets with each sub-mesh instantiating a unique Receptacle. * update receptacle names in test for new code * refactor to use Meshdata internally * convert triangle-like primitives to triangles. * refactor to use Receptacle unique_name for sample filtering. Add support for filter files configured in scene instance config user_defined fields. * refactor snapping functionality to improve stability by estimating consistency of support surface from bb corners projected in gravity direction * Update habitat-lab/habitat/sims/habitat_simulator/sim_utilities.py Co-authored-by: Aaron Gokaslan <[email protected]> * turn off additional stability culling for now * bugfix * fix margin issues * Add TODO comment * Misc changes to work with merge * Add missing import * Fix target sampling bug * Remove unneeded code * Add changes to test previously missed --------- Co-authored-by: Alexander Clegg <[email protected]> Co-authored-by: Aaron Gokaslan <[email protected]>

…research#1108) * updates to rearrange episode generation to support mesh receptacles and improved debugging * docstrings and typing for DebugVisualizer util * mesh receptacle debug_draw with global transforms * add some tests for sampling accuracy * bugfix - use default_sensor_uid instead of hardcoded "rgb" in DebugVisualizer * adjust debug peek target to bb center to better capture objects not centered at COM * add debug circles to the peek API * add peek_scene variant for easliy getting images of a full scene or stage.

aclegg3 added 10 commits January 30, 2023 09:34

updates to rearrange episode generation to support mesh receptacles a…

9ba9854

…nd improved debugging

add some comments

307bcc5

docstrings and typing

9718a8b

docstrings and typing for DebugVisualizer util

352065e

update config to test against habitat-sim branch with updated test as…

2ed69ef

…sets

bugfix with global transform + cleanup

492461d

mesh receptacle debug_draw with global transforms

61de77b

add test WIP

9344616

add some tests for sampling accuracy

1bc1aad

remove erroneous comment

ecb498d

aclegg3 requested review from ASzot, vincentpierre and rpartsey February 1, 2023 00:35

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Feb 1, 2023

aclegg3 commented Feb 1, 2023

View reviewed changes

Merge remote-tracking branch 'origin/main' into rearrange-gen-improve…

bde1c2a

…ments

aclegg3 mentioned this pull request Feb 1, 2023

[Example] - Receptacle viewer facebookresearch/habitat-sim#1995

Closed

11 tasks

aclegg3 added 5 commits February 1, 2023 09:52

pre-commit changes

8ca69f4

numpy typing fix

768645a

move triangle sampling and testing logic to geometry_utils and move/a…

c9d5eea

…dd tests

Merge remote-tracking branch 'origin/main' into rearrange-gen-improve…

5afe555

…ments

formatting from black 23.1

28ee901

aclegg3 mentioned this pull request Feb 10, 2023

Update magnum-bindings for MeshData fix on py3.7 facebookresearch/habitat-sim#2003

Merged

11 tasks

use magnum trade importer

59fecd1

mosra reviewed Feb 10, 2023

View reviewed changes

aclegg3 added 4 commits February 10, 2023 13:26

use AnySceneImporter

d1f49e4

try fixing the plugin import issue

9a0c7ad

Merge remote-tracking branch 'origin/main' into rearrange-gen-improve…

dc2b5ab

…ments

pre-commit

b7a0a92

vincentpierre reviewed Feb 27, 2023

View reviewed changes

aclegg3 added 5 commits March 7, 2023 17:54

bugfix - use default_sensor_uid instead of hardcoded "rgb"

f6b9a82

adjust debug peek target to bb center to better capture objects not c…

b378939

…entered at COM

add debug circles to the peek API

c0c4cb0

address review comments

51a4d14

refactor stability logging per review comments

13c8770

aclegg3 requested a review from vincentpierre March 11, 2023 19:11

vincentpierre approved these changes Mar 13, 2023

View reviewed changes

add peek_scene variant for easliy getting images of a full scene or s…

1695371

…tage.

aclegg3 requested a review from mosra March 13, 2023 21:45

mosra approved these changes Mar 14, 2023

View reviewed changes

aclegg3 added 4 commits March 14, 2023 15:49

revert testing on sim branch back to main

226a829

Merge remote-tracking branch 'origin/main' into rearrange-gen-improve…

e49fc80

…ments

Add a note and todo about stability culling flag

8d34540

Remove unused config file

1c9fb5d

aclegg3 merged commit 0ef0de2 into main Mar 15, 2023

aclegg3 deleted the rearrange-gen-improvements branch March 15, 2023 16:36

0mdc mentioned this pull request Mar 16, 2023

[CI Fix] Fix typing error. #1203

Merged

5 tasks

JHurricane96 mentioned this pull request May 8, 2023

OVMM: Merge episode generation updates from main #1313

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TriangleMeshReceptacles and RearrangeGenerator Improvements #1108

TriangleMeshReceptacles and RearrangeGenerator Improvements #1108

aclegg3 commented Feb 1, 2023 •

edited

Loading

aclegg3 Feb 1, 2023

mosra Feb 10, 2023

mosra Feb 10, 2023

aclegg3 Feb 10, 2023

mosra Feb 11, 2023

vincentpierre left a comment

vincentpierre Feb 27, 2023

mosra Mar 3, 2023

aclegg3 Mar 11, 2023

aclegg3 Mar 14, 2023

aclegg3 commented Mar 11, 2023

vincentpierre left a comment

vincentpierre Mar 13, 2023

aclegg3 Mar 13, 2023

TriangleMeshReceptacles and RearrangeGenerator Improvements #1108

TriangleMeshReceptacles and RearrangeGenerator Improvements #1108

Conversation

aclegg3 commented Feb 1, 2023 • edited Loading

Motivation and Context

How Has This Been Tested

Types of changes

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vincentpierre left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aclegg3 commented Mar 11, 2023

vincentpierre left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aclegg3 commented Feb 1, 2023 •

edited

Loading