[Bug Report] Increasing the number of parallel environments results in severe degradation in TiledCamera rendering quality

[trevormcinroe]

I am using the latest version (4.2.0) on a Ubuntu server while rendering headless.

Here is the quality of images rendered from one camera while only using a single environment:

Here is the quality of images rendered from one camera while using 32 parallel environments:

[glvov-bdai]

Hi can you please share the script /conditions to recreate these results?

[glvov-bdai]

In my preliminary tests I did not see any degradation with 32 or 64 envs in parallel. However, I only tried low resolution and static frames. If you provide system information (how did you install, GPU, ubuntu version, etc) and a way for us to recreate your experiment I'd be happy to look into this further! (see bottom for my recreation steps)

32 envs

64 envs

I tried these with the manager based cartpole envs. You can recreate my results from my fork and the binary install with

git clone https://github.com/glvov-bdai/IsaacLab.git sanity_check
cd sanity_check && git checkout write_tiled_rendering
# COPY OVER SIM_LINK TO _isaac_sim # For example: ln -s /home/nvidia/.local/share/ov/pkg/isaac-sim-4.2.0 _isaac_sim
./isaaclab.sh -p source/standalone/workflows/rl_games/train.py --task Isaac-Cartpole-RGB-Camera-v0 --num_envs 32 --enable_cameras

Numbered images will be saved to parent directory

[glvov-bdai]

I did notice that the grid lines do seem to shift step to step

[trevormcinroe]

@glvov-bdai I can provide some minimal code tomorrow, but try with a more visually-complex model like Franka, and place the camera closer to the object. Perhaps that will show the issue?

[StoneT2000]

I noticed that some environments (i guess the ones that spawn too far away?) dont show the floor anymore. My understanding is Isaac Lab essentially has one shared scene meaning objects can fly into the parallel views of others if they move there and technically there is only one floor. Is there a way to make the floor bigger so that all parallel cameras have a floor shown? I am also curious if this may cause changes in rendering performance if some parallel envs are / are not rendering the floor

[glvov-bdai]

@StoneT2000 We are aware of the default ground plane not being visible beyond a certain contained size.
This is due to the ground plane being a finite USD that is not scaled. If you'd like, you could add more ground planes as assets, and just spawn them further away. Alternatively, you could use a larger USD asset for the ground plane, just having it span the desired sizes.

Although the default ground plane may not be visible beyond certain sizes, it should still present for collisions.

In terms of rendering performance, you can try my benchmark tool #976 with the different configurations I suggested and find empirically what works best.

If you run into any problems with the ground plane stuff I'd recommend opening an issue specifically for that for better visibility - doesn't seem like it's related to the rendering issues

[rthaker01]

Hi @trevormcinroe - Would you please share the repro steps?

[kellyguo11]

The rendering quality difference could be coming from the denoiser in the renderer. One thing you could try is adding this to the code to change the denoiser mode to see if it'll help:

import omni.replicator.core as rep
rep.settings.set_render_rtx_realtime(antialiasing="DLAA")

[trevormcinroe]

@rthaker01 @glvov-bdai apologies for the delay. ICLR deadline and all... I've been working with @elle-miller on some projects using IsaacLab.

We've both been opening bug reports, and we'll get back to you soon on this one with more information.

[elle-miller]

Hi @glvov-bdai @kellyguo11!

I have created a minimal working example repo based on most recent Isaac Lab + skrl version here: https://github.com/elle-miller/camera_testing/tree/main

Running the below commands will create image folders that save the raw tiled camera output for 100 steps. I have uploaded several folders with different num_envs

python scripts/train.py --task ImageCartpole --headless --enable_cameras --num_envs 4
python scripts/train.py --task ImageCartpole --headless --enable_cameras --num_envs 128

The effects have been more noticeable in more complex environments e.g. franka lift, but I think you can still see the degradation with cartpole.

1 env

1024 envs

It appears that @kellyguo11's suggestion fixes this problem:

without denoiser

with denoiser

I will double check this fix with more complex environments. But it appears that the denoiser should be the default setting.

[ArneKlages4444]

Hi, I tested the denoiser fix in a visually more complex environment. It seems that the denoiser helps a lot but does not solve the problem completely, see the following images (112x112 pixel). Especially when it comes to lighting, the degeneration effect is still very visible.

2 envs:

1024 envs with denoiser:

1024 envs:

2 envs:

1024 envs with denoiser:

1024 envs:

System:

• IsaacLab version (last commit Sep 20, 2024): Using Orbit in Isaac Sim Extension Workflows #132
• Isaac Sim Version: 4.2.0
• GPU: L40S (48 GiB)
• CUDA Version: 12.2
• GPU Driver Version: 535.161.08

[elle-miller]

@glvov-bdai OK, while the denoiser helps, the fix still does not make images in a good state for training (84px images, 1024 envs). The strange trailing effect of the cart still persists. Has anyone else experienced this?

Without denoiser:

With denoiser:

[glvov-bdai]

Hi @elle-miller @ArneKlages4444 @trevormcinroe

Thank you for investigating this issue and providing replications steps.

@kellyguo11 @mpgussert @Dhoeller19 still looks like the issue is persisting on isaac sim side?

Can we please add this to functionality to fix in the next isaac release?

[SantiDiazC]

Hi,
I have experienced a similar issue when modifying the pose of the camera to render two different views.
In this case the image has some degradation when I do that, and it get worse when the number of environments are increased. Also, I changed the denoiser as @kellyguo11 suggested. Is there any solution for this? Thank you!

I am using the latest isaaclab 1.4.0 and Isaacsim 4.2.

[ArneKlages4444]

It seems that with Isaac Sim 4.5.0 and Isaac Lab 2.0.0, this problem is partially solved. I can't notice any image degradation when using a high number of environments.

DLAA 10 envs:

DLAA 1020 envs:

DLAA 4095 envs:

Default config 10 envs:

Default config 1020 envs:

The issue with the ghosting (trailing effect) mentioned by @elle-miller still remains. I think this is a problem of DLSS and DLAA, because they rely on previous frames. If you turn antialiasing off or use FXAA (that does not use previous frames), the ghosting is gone. But the problem with this approach is that the image quality suffers. I still prefer using DLAA with low image resolutions.

Off

FXAA

[kellyguo11]

Thanks for confirming! We are working with our rendering team on ways to reduce the ghosting effect, it is indeed likely due to DLSS.