[Question] Problems with a high number of environments when using Cameras #604
Comments
|
Just another user here. Maybe give |
|
Yes, the "When using the |
|
In a large number of camera simulation scenarios, I also encountered a similar error, is your problem solved? |
|
No, sadly, it's not solved yet. The only way to increase the number of environments that I found is to add a second GPU. But the increase in parallel environments is not linear with the amount of separate GPUs. So it's up to you if it's worth it, especially because only a fraction of the GPU's VRAM capacity is used. Using more than two GPUs did not help in my tests (I tested it with up to 4 L40S GPUs). |
|
I got same warning and my simulation freezes with 256 environments on my A6000 machine. |
|
Tiled rendering RGB has been improved in the 1.2 release, which fixes #493 I am able to simulate 2048 low resolution tiled cameras on a laptop 3080. We are keeping an eye on #1031 You can try many cameras with your task environment https://isaac-sim.github.io/IsaacLab/source/how-to/estimate_how_many_cameras_can_run.html . If you still experience this issue with tiled rendering let us know! |
We encounter issues with a high number of environments when using Cameras. Although the memory problem with cameras is now fixed in Isaac Lab, we cannot scale up the amount of environments as we like.
When using two graphic cards, we could achieve a parallelization of 475 environments in the test setup. When going up to 476, the setup sometimes works, or we get one of two errors (see below, errors 1 and 2). With more than 476 environments (tested up to 512), the tests always fail with error 1. When using only one graphics card, we could not even achieve 475 environments and got error 3. The memory usage was tracked while we were performing the experiments, and there was always plenty of VRAM left (see memory usage with 475). We did these tests on a Linux machine with TITAN RTX GPUs and reproduced them on our HPC with L40S GPUs. On both systems, the breaking points were the same, despite the more powerful L40S GPUs.
For our test, we simply put a camera into the
FrankaCubeLiftenvironment and added the flattened image to the observations (code below).Is there anything we do wrong in our setups, or is this a known issue with cameras? When using the
TiledCameracamera, we could use 4096 parallel environments, but since it does not provide the desired image output (#493) it is not usable for us.System Info:
System 1:
System 2 (HPC):
Memory usage with 475:
Error 1 (two GPUs and num_envs > 476):
Error 2 (two GPUs and sometimes at num_envs = 476):
Error 3 (one L40S and 475 environments):
Test Code:
Console command:
./isaaclab.sh -p test_fr.py --headless --enable_cameras --num_envs=476The text was updated successfully, but these errors were encountered: