Skip to content

[Bug Report] Multiple Tiled Cameras Override Each Other #1070

Closed
@ori-gadot

Description

@ori-gadot

Bug Description

When using two tiled cameras in the scene, the last camera overrides all observations, resulting in both cameras capturing the same image.

Steps to reproduce

  1. Run the following script, which initializes the CartpoleRGBCameraEnvCfg environment with two environments and two tiled cameras.
from omni.isaac.lab.app import AppLauncher

app_launcher = AppLauncher(headless=False, enable_cameras=True)
simulation_app = app_launcher.app

"""Rest everything follows."""

import torch

from omni.isaac.lab.envs import ManagerBasedRLEnv
from omni.isaac.lab.sensors.camera.utils import save_images_to_file

from omni.isaac.lab_tasks.manager_based.classic.cartpole import CartpoleRGBCameraEnvCfg


def main():
    env_cfg = CartpoleRGBCameraEnvCfg()
    env_cfg.scene.num_envs = 2
    env = ManagerBasedRLEnv(cfg=env_cfg)

    while simulation_app.is_running():
        with torch.inference_mode():
            actions = torch.rand_like(env.action_manager.action)
            obs, _, _, _, _ = env.step(actions)
            save_images_to_file(obs["policy"]["image1"], "/tmp/image1.png")
            save_images_to_file(obs["policy"]["image2"], "/tmp/image2.png")
    env.close()


if __name__ == "__main__":
    main()
    simulation_app.close()
  1. Modify the CartpoleRGBCameraSceneCfg class in the following file: source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/manager_based/classic/cartpole/cartpole_camera_env_cfg.py, adding two tiled cameras as shown below:
@configclass
class CartpoleRGBCameraSceneCfg(CartpoleSceneCfg):
    tiled_camera1: TiledCameraCfg = TiledCameraCfg(
        prim_path="{ENV_REGEX_NS}/Camera1",
        offset=TiledCameraCfg.OffsetCfg(pos=(-7.0, 0.0, 3.0), rot=(0.9945, 0.0, 0.1045, 0.0), convention="world"),
        data_types=["rgb"],
        spawn=sim_utils.PinholeCameraCfg(
            focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.1, 20.0)
        ),
        width=1280,
        height=720,
    )

    tiled_camera2: TiledCameraCfg = TiledCameraCfg(
        prim_path="{ENV_REGEX_NS}/Camera2",
        offset=TiledCameraCfg.OffsetCfg(pos=(-7.0, 0.0, 3.0), convention="world"),
        data_types=["rgb"],
        spawn=sim_utils.PinholeCameraCfg(
            focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.1, 20.0)
        ),
        width=1280,
        height=720,
    )
  1. Modify RGBObservationsCfg to include observations for both cameras:
@configclass
class RGBObservationsCfg:
    """Observation specifications for the MDP."""

    @configclass
    class RGBCameraPolicyCfg(ObsGroup):
        """Observations for policy group."""

        image1 = ObsTerm(func=grab_images, params={"sensor_cfg": SceneEntityCfg("tiled_camera1"), "data_type": "rgb", "normalize": True})
        image2 = ObsTerm(func=grab_images, params={"sensor_cfg": SceneEntityCfg("tiled_camera2"), "data_type": "rgb", "normalize": True})

        def __post_init__(self) -> None:
            self.enable_corruption = False
            self.concatenate_terms = False

    policy: ObsGroup = RGBCameraPolicyCfg()
  1. Modify the scene lights in source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/manager_based/classic/cartpole/cartpole_env_cfg.py for clearer image results:
@configclass
class CartpoleSceneCfg(InteractiveSceneCfg):
    """Configuration for a cart-pole scene."""

    # ground plane
    ground = AssetBaseCfg(
        prim_path="/World/ground",
        spawn=sim_utils.GroundPlaneCfg(size=(100.0, 100.0), color=(0.5, 0.5, 0.5)),
    )

    # cartpole
    robot: ArticulationCfg = CARTPOLE_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")

    # lights
    dome_light = AssetBaseCfg(
        prim_path="/World/DomeLight",
        spawn=sim_utils.DomeLightCfg(color=(0.9, 0.9, 0.9), intensity=10.0),
    )
    distant_light = AssetBaseCfg(
        prim_path="/World/DistantLight",
        spawn=sim_utils.DistantLightCfg(color=(0.9, 0.9, 0.9), intensity=1.0),
        init_state=AssetBaseCfg.InitialStateCfg(rot=(0.738, 0.477, 0.477, 0.0)),
    )

    sphere_light = AssetBaseCfg(
        prim_path="{ENV_REGEX_NS}/SphereLight",
        spawn=sim_utils.SphereLightCfg(
            color=(0.1, 0.1, 0.1), enable_color_temperature=True, color_temperature=5500, intensity=100, radius=2.0, exposure=10.0,
        ),
        init_state=AssetBaseCfg.InitialStateCfg(pos=(0, 0, 5)),
    )

Additional context

Two identical tiled cameras with slightly different orientations (one capturing more of the ground, the other more of the sky) were used. Observation data was saved as images for inspection.

Running the following scenarios, only the first one reproduces the bug:

  • Scenario 1: Running two tiled cameras in two environments causes the bug—both image files show the second camera’s view.

    • Camera 1
      image1

    • Camera 2
      image2

  • Scenario 2: Running two tiled cameras in a single environment works correctly—each camera renders its own FOV.

    • Camera 1
      image1

    • Camera 2
      image2

  • Scenario 3: Running two non-tiled cameras in two environments works as expected.

    • Camera 1
      image1

    • Camera 2
      image2

Note: Although it may be unrelated, there is a noticeable decline in image quality in the first scenario compared to the other two.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or requestisaac-simThis issue is related to Isaac Sim

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions