Added updates for HM3D ObjectNav training #818

srama2512 · 2022-02-28T14:47:21Z

Motivation and Context

This PR adds necessary configs for HM3D ObjectNav training. It includes new features for training with a SemanticCategory sensor. It also updates the generate_video function to include scene_id, goal_name in the save path.

How Has This Been Tested

Tested locally.

Types of changes

Docs change / refactoring / dependency upgrade
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have completed my CLA (see CONTRIBUTING)
I have added tests to cover my changes.
All new and existing tests passed.

habitat/tasks/nav/nav.py

Skylion007 · 2022-03-01T18:29:41Z

habitat/tasks/nav/nav.py

+    def get_observation(
+        self, *args: Any, observations, episode, **kwargs: Any
+    ):
+        episode_uniq_id = f"{episode.scene_id} {episode.episode_id}"


The semantic sensor oservation here could be a Tensor right, which would create an unnecessary CPU <-> GPU. I think you can easily modify this to operator on both NpArrays and Torch.Tensor

this is caching.

Skylion007 · 2022-03-01T18:30:35Z

@srama2512 This needs testing in the test suite like from so:

habitat-lab/test/test_sensors.py

Line 443 in 3f2c7cb

"sensor", ["RGB_SENSOR", "DEPTH_SENSOR", "SEMANTIC_SENSOR"]

mathfac · 2022-03-01T23:11:53Z

habitat_baselines/rl/ddppo/policy/resnet_policy.py

+        if "semantic_category" in observation_space.spaces:
+            self._n_input_semantic_category = observation_space.spaces["semantic_category"].shape[2]
+            spatial_size = observation_space.spaces["semantic_category"].shape[0] // 2
+            assert self._n_input_semantic_category == 3, "ResNetEncoder only supports RGB values from SemanticCategory sensor!"


Why semantic_category why it has RGB values, not int id of the category?

Right now, the semantic category sensor outputs an RGB image by default. For each goal category, there is a unique color. For all other categories and backgrounds, the pixels are set to zeros.

mathfac · 2022-03-01T23:12:23Z

habitat_baselines/rl/ddppo/policy/resnet_policy.py

        if normalize_visual_inputs:
            self.running_mean_and_var: nn.Module = RunningMeanAndVar(
-                self._n_input_depth + self._n_input_rgb
+                self._n_input_depth + self._n_input_rgb + self._n_input_semantic_category


normalizing semantic_category inputs doesn't look right.

The semantic_category input is forced to be RGB (see earlier comment).

mathfac · 2022-03-01T23:15:18Z

habitat_baselines/rl/ppo/ppo_trainer.py

@@ -66,6 +66,7 @@ class PPOTrainer(BaseRLTrainer):
    supported_tasks = ["Nav-v0"]

    SHORT_ROLLOUT_THRESHOLD: float = 0.25
+    SENSORS_BLACKLIST = ["semantic"]


That looks hack to do it in PPOTrainer. Why we need to disable this sensor here?

Having the semantic value throws errors in various parts of the code (observation transforms, observations rollouts, etc). You could try removing it from the blacklist and checking. This seemed the more elegant solution, but I'm happy to implement a better solution based on feedback.

mathfac · 2022-03-01T23:17:40Z

habitat_baselines/rl/ppo/ppo_trainer.py

@@ -400,6 +406,15 @@ def _extract_scalars_from_info(

        return result

+    def _clean_observations(self, observations):


We may need to do it different way.

mathfac · 2022-03-01T23:21:59Z

habitat_baselines/rl/ppo/ppo_trainer.py

@@ -1058,16 +1076,21 @@ def _eval_checkpoint(
                            current_episodes[i].episode_id,
                        )
                    ] = episode_stats
+                    goal_name = None


That's too specific and should live in observations_to_image as it has access to current_episodes[i].

The idea was to add the goal name to the video save path (so that we can visualize episodes based on the goal). How do you suggest we go about this? Can we pass the current_episodes[i] to generate_video and move the naming logic there?

mathfac · 2022-03-01T23:22:21Z

habitat_baselines/rl/ppo/ppo_trainer.py

                            episode_id=current_episodes[i].episode_id,
                            checkpoint_idx=checkpoint_index,
                            metrics=self._extract_scalars_from_info(infos[i]),
                            tb_writer=writer,
+                            goal_name=goal_name,


In high level there should be no task specific logic in this file.

mathfac · 2022-03-02T05:26:42Z

habitat/tasks/nav/nav.py

+            cat_mapping = HM3D_CATEGORY_TO_TASK_CATEGORY_ID
+        self.category_to_task_category_id = cat_mapping
+        if config.RAW_NAME_TO_CATEGORY_MAPPING != "":
+            with open(config.RAW_NAME_TO_CATEGORY_MAPPING, "r") as fp:


If we leave this code it should use some csv reader or etc.

mathfac · 2022-03-02T05:32:08Z

habitat/tasks/nav/nav.py

@@ -514,6 +521,118 @@ def get_observation(
        )


+@registry.register_sensor(name="SemanticCategorySensor")
+class SemanticCategorySensor(Sensor):


Do we use this sensor for training or only for visualization purposes?

This sensor is used for training models with GT semantic goal inputs.

@srama2512, yes. But that GT data isn't available during evaluation and from this PR isn't clear how it will be replaced.

mathfac · 2022-03-02T05:47:07Z

habitat/tasks/nav/nav.py

+        # Map from instance id to task id
+        semantic_category = np.take(self.instance_id_to_task_id, semantic)
+        if self.config.CONVERT_TO_RGB:
+            semantic_category = self.convert_semantic_to_rgb(semantic_category)


RGB conversion should live in def observations_to_image(observation: Dict, info: Dict) -> np.ndarray:.
Something like this:

if "semantic_category" in observation: flat_sem = observation[ obj_semantic_name ] # to move to same scale #.permute(2, 0, 1).unsqueeze(0).data.max(1)[1].cpu().numpy()[0] flat_sem[flat_sem == 41] = 40 if not isinstance(flat_sem, np.ndarray): flat_sem = flat_sem.cpu().numpy() semantic_segmentation = ( color_label(flat_sem).squeeze().transpose(1, 2, 0).astype(np.uint8) ) egocentric_view.append(semantic_segmentation) if "objectgoal" in observation and "episode_info" in info: from habitat.tasks.nav.object_nav_task import task_cat2mpcat40 # permute tensor to dimension [CHANNEL x HEIGHT X WIDTH] idx = task_cat2mpcat40[observation["objectgoal"][0]] goal_segmentation = ( color_label(flat_sem == idx) .squeeze() .transpose(1, 2, 0) .astype(np.uint8) ) egocentric_view.append(goal_segmentation)

This sensor is not for visualization purposes. It is intended to be a model input for training with oracle semantics. So the logic should live within the sensor itself, right?

mathfac · 2022-03-02T05:49:30Z

habitat/tasks/nav/semantic_constants.py

+# LICENSE file in the root directory of this source tree.
+
+
+GIBSON_CATEGORY_TO_TASK_CATEGORY_ID = {


This file should be somewhere: habitat/datasets/object_nav/....

Co-authored-by: Aaron Gokaslan <[email protected]>

erikwijmans · 2022-03-03T22:15:56Z

habitat/tasks/nav/semantic_constants.py

+}
+
+
+MP3D_CATEGORY_TO_TASK_CATEGORY_ID = {


This is different than https://github.com/niessner/Matterport/blob/master/metadata/mpcat40.tsv which will likely lead to confusion.

erikwijmans · 2022-03-03T22:16:27Z

habitat/tasks/nav/semantic_constants.py

+
+
+GIBSON_CATEGORY_TO_TASK_CATEGORY_ID = {
+    'chair': 0,


Based on "Set invalid instance IDs to unknown object 0" you use 0 as unknown elsewhere.

mathfac

That's still not clear why we need Semantic_category represent as RGB for training. Non-intuitive and can lead to more issues when categories are cross dataset.
As it uses GT Semantic information it's not clear how this PR is used to train final baseline.

mathfac · 2022-08-30T20:37:51Z

@srama2512, @devendrachaplot how we want evolve this PR? Do we more up to date code focused on ObjectNav

srama2512 added 5 commits February 28, 2022 06:17

added missing HM3D pointnav config

c04d8f3

added semantic category sensor and hm3d configs

8e11f2d

added scene_id, goal_name to generate_video

619845b

updated hm3d objectnav in README

6a13829

added missing hm3d dataset yaml

9638fa7

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Feb 28, 2022

Skylion007 reviewed Mar 1, 2022

View reviewed changes

habitat/tasks/nav/nav.py Outdated Show resolved Hide resolved

Skylion007 reviewed Mar 1, 2022

View reviewed changes

mathfac reviewed Mar 1, 2022

View reviewed changes

mathfac reviewed Mar 2, 2022

View reviewed changes

Update habitat/tasks/nav/nav.py

0c4a22d

Co-authored-by: Aaron Gokaslan <[email protected]>

erikwijmans reviewed Mar 3, 2022

View reviewed changes

mathfac reviewed Mar 8, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added updates for HM3D ObjectNav training #818

Added updates for HM3D ObjectNav training #818

srama2512 commented Feb 28, 2022 •

edited

Loading

Skylion007 Mar 1, 2022

mathfac Mar 2, 2022

Skylion007 commented Mar 1, 2022

mathfac Mar 1, 2022

srama2512 Mar 3, 2022

mathfac Mar 1, 2022

srama2512 Mar 3, 2022

mathfac Mar 1, 2022

srama2512 Mar 3, 2022

mathfac Mar 1, 2022

mathfac Mar 1, 2022

srama2512 Mar 3, 2022 •

edited

Loading

mathfac Mar 1, 2022

mathfac Mar 2, 2022

mathfac Mar 2, 2022

srama2512 Mar 3, 2022

mathfac Mar 8, 2022

mathfac Mar 2, 2022

srama2512 Mar 3, 2022

mathfac Mar 2, 2022

erikwijmans Mar 3, 2022

erikwijmans Mar 3, 2022

mathfac left a comment

mathfac commented Aug 30, 2022

		@@ -400,6 +406,15 @@ def _extract_scalars_from_info(

		return result

		def _clean_observations(self, observations):

		# LICENSE file in the root directory of this source tree.


		GIBSON_CATEGORY_TO_TASK_CATEGORY_ID = {

		}


		MP3D_CATEGORY_TO_TASK_CATEGORY_ID = {

Added updates for HM3D ObjectNav training #818

Are you sure you want to change the base?

Added updates for HM3D ObjectNav training #818

Conversation

srama2512 commented Feb 28, 2022 • edited Loading

Motivation and Context

How Has This Been Tested

Types of changes

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Skylion007 commented Mar 1, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srama2512 Mar 3, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mathfac left a comment

Choose a reason for hiding this comment

mathfac commented Aug 30, 2022

srama2512 commented Feb 28, 2022 •

edited

Loading

srama2512 Mar 3, 2022 •

edited

Loading