multiple sensors

[cjmdd]

How to add multiple sensors (multi view camera and LiDAR)? How to establish a connection with Isaac gym/sim ixi

[alex-petrenko]

Hi @cjmdd
Handling multiple sensory inputs is relatively straightforward.

1. the environment observation space needs to be a dict. Multiple keys in the dict are responsible for different sensors. There are many envs that implement dictionary observation spaces like this, VizDoom and DMlab are some examples
2. You will need a custom model (custom "encoder" specifically) to encode multiple sensory inputs. Some docs on custom models here: https://www.samplefactory.dev/03-customization/custom-models/
3. See these docs for isaacgym integration https://www.samplefactory.dev/09-environment-integrations/isaacgym/

For some inspiration, here's a project where we trained a Doom agent with up to three sensory inputs:

• RGB images (pixels)
• audio signal (sounds from the game)
• in-game data like health, ammo, etc.