[Question] Dynamically replace objects in an environment during simulation

[virgilio96upm]

Question

There is a way to replace objects spawned in the environment while the simulation is running? (spawn a .usd of a new object and remove the one already defined in the configuration of the environment after the simulation completes a set of iterations or a certain condition is met).

So far I have tested that I can successfully add multiple usd into a manager-based RL example of the Franka Lift task example but I would like to know if it is possible to update the objects while the simulation is running.

# Copyright (c) 2022-2025, The Isaac Lab Project Developers.
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

from omni.isaac.lab.assets import RigidObjectCfg
from omni.isaac.lab.sensors import FrameTransformerCfg
from omni.isaac.lab.sensors.frame_transformer.frame_transformer_cfg import OffsetCfg
from omni.isaac.lab.sim.schemas.schemas_cfg import RigidBodyPropertiesCfg
from omni.isaac.lab.sim.spawners.from_files.from_files_cfg import UsdFileCfg
from omni.isaac.lab.utils import configclass
from omni.isaac.lab.utils.assets import ISAAC_NUCLEUS_DIR

## additional module added for multiple usd spawning and randomization
from omni.isaac.lab.sim.spawners.wrappers.wrappers_cfg import MultiUsdFileCfg



from omni.isaac.lab_tasks.manager_based.manipulation.lift import mdp
from omni.isaac.lab_tasks.manager_based.manipulation.lift.lift_env_cfg import LiftEnvCfg

##
# Pre-defined configs
##
from omni.isaac.lab.markers.config import FRAME_MARKER_CFG  # isort: skip
from omni.isaac.lab_assets.franka import FRANKA_PANDA_CFG  # isort: skip

@configclass
class FrankaCubeLiftEnvCfg(LiftEnvCfg):
    def __post_init__(self):
        # post init of parent
        super().__post_init__()
        self.scene.replicate_physics= False
        # Set Franka as robot
        self.scene.robot = FRANKA_PANDA_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")

        # Set actions for the specific robot type (franka)
        self.actions.arm_action = mdp.JointPositionActionCfg(
            asset_name="robot", joint_names=["panda_joint.*"], scale=0.5, use_default_offset=True
        )
        self.actions.gripper_action = mdp.BinaryJointPositionActionCfg(
            asset_name="robot",
            joint_names=["panda_finger.*"],
            open_command_expr={"panda_finger_.*": 0.04},
            close_command_expr={"panda_finger_.*": 0.0},
        )
        # Set the body name for the end effector
        self.commands.object_pose.body_name = "panda_hand"
        # Set tool as object
        self.scene.object = RigidObjectCfg(
            prim_path="{ENV_REGEX_NS}/Object",
            init_state=RigidObjectCfg.InitialStateCfg(pos=[0.4, 0, 0.055], rot=[0, 0, 0, 1]),
            ########## UPDATE THIS LIST OF USD FILES DURING SIMULATION TIME##############
            spawn=MultiUsdFileCfg(
                usd_path= [
                    f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/DexCube/dex_cube_instanceable.usd",
                    f"{ISAAC_NUCLEUS_DIR}/Props/Blocks/MultiColorCube/multi_color_cube_instanceable.usd",
                ],
                random_choice=True,
                scale=(0.8, 0.8, 0.8),
                rigid_props=RigidBodyPropertiesCfg(
                    solver_position_iteration_count=16,
                    solver_velocity_iteration_count=1,
                    max_angular_velocity=1000.0,
                    max_linear_velocity=1000.0,
                    max_depenetration_velocity=5.0,
                    disable_gravity=False,
                ),
            ),
        )

        # Listens to the required transforms
        marker_cfg = FRAME_MARKER_CFG.copy()
        marker_cfg.markers["frame"].scale = (0.1, 0.1, 0.1)
        marker_cfg.prim_path = "/Visuals/FrameTransformer"
        self.scene.ee_frame = FrameTransformerCfg(
            prim_path="{ENV_REGEX_NS}/Robot/panda_link0",
            debug_vis=False,
            visualizer_cfg=marker_cfg,
            target_frames=[
                FrameTransformerCfg.FrameCfg(
                    prim_path="{ENV_REGEX_NS}/Robot/panda_hand",
                    name="end_effector",
                    offset=OffsetCfg(
                        pos=[0.0, 0.0, 0.1034],
                    ),
                ),
            ],
        )


@configclass
class FrankaCubeLiftEnvCfg_PLAY(FrankaCubeLiftEnvCfg):
    def __post_init__(self):
        # post init of parent
        super().__post_init__()
        # make a smaller scene for play
        self.scene.num_envs = 50
        self.scene.env_spacing = 2.5
        # disable randomization for play
        self.observations.policy.enable_corruption = False

[glvov-bdai]

I'd recommend trying https://isaac-sim.github.io/IsaacLab/main/source/how-to/multi_asset_spawning.html for this use case

[RandomOakForest]

Thank you for posting this. In general, the team recommends to not replace objects during simulation.

[StrainFlow]

This is a fairly common need and I'm going to try to figure out a way to to this that works reasonably well. Sometimes you just need to randomize the rigid bodies in an environment.

To be clear, I'll be detecting if a robot in an environment has completed its task, resetting its environment if it has and doing this as the environments reset, not mid-episode

[virgilio96upm]

Sounds like a great approach! Have you had a chance to test it yet? I'd be interested to hear how well it works in practice.

[StrainFlow]

Yep, works great :)

[StrainFlow]

2025-03-10.09-15-30.mp4

[virgilio96upm]

That looks great! Are you randomizing the pose of the objects within the environment, or are you spawning entirely new objects into the simulation after each episode? In the video, it looks like the previous object or its pose fades out until it disappears.

In my case, I would like to replace the spawned object with a new one after the episode finishes. How are you handling object replacement in your setup?

Thank you again for your response!

[bamertsa]

I have a very similar use case. As a solution, I spawn a randomized object pool for every environment. Whenever an environment is reset, objects are randomly chosen from the object pool. I tried to illustrate what happens at every reset in the image below.

For all the objects that are not needed, the positions are set to something like (100, 100, 1).

Unfortunately, this can have quite a performance impact for large object pools when all objects have collision enabled. So far, I haven't found a satisfactory solution. I am not sure if there is a possibility to disable/enable collision checks for objects during runtime. I experimented with my_rigid_object_collection.root_physx_view.set_disable_simulations() and similar functions to no avail. Maybe someone here has a good suggestion?

[Axellwppr]

Hi, I'm having a similar issue—neither set_disable_gravities nor set_disable_simulations worked for me. Did you find another solution? Thanks!

[bamertsa]

Unfortunately I did not. I have all the collision checks and other simulation aspects enabled for now.

[XiRouz]

Great solution, @bamertsa, I have thought of an object pool as well! Do you mind answering some questions about your approach?
Do you somehow clone objects from that pool on the scene and put them in your environment, or do you move the objects from where the pool resides, to your environment, and then move them back to the pool?
Do you have to reset the environment to make a new set of objects in environment, or you can change them without resetting the environment?