[RLlib][Unity3D] multi_agent_episode.py raised exception : did NOT receive a last (truncation) observation

[Ginurx]

What happened + What you expected to happen

The following two expections in multi_agent_episode.py can be raised when training with the Unity3DEnv.

ray/rllib/env/multi_agent_episode.py

Lines 543 to 547 in 29e6029

	raise MultiAgentEnvError(
	f"Agent {agent_id} acted and then got truncated, but did NOT "
	"receive a last (truncation) observation, required for e.g. "
	"value function bootstrapping!"
	)

ray/rllib/env/multi_agent_episode.py

Lines 587 to 592 in 29e6029

	if _truncated:
	raise MultiAgentEnvError(
	f"Agent {agent_id} acted and then got truncated, but did "
	"NOT receive a last (truncation) observation, required "
	"for e.g. value function bootstrapping!"
	)

This issue occurs because the ML-Agents' UnityEnvironment.get_steps(...) function doesn't always return steps of all the agents at every timestep.

Example:

Consider the 3DBall environment in ML-Agents , which has 12 agents in the scene.

'env_runners': 
  {
  'num_agent_steps_sampled': 
    {'3DBall?team=0_0': 140,
    '3DBall?team=0_1': 140,
    '3DBall?team=0_10': 139,
    '3DBall?team=0_11': 137,
    '3DBall?team=0_2': 141,
    '3DBall?team=0_3': 141,
    '3DBall?team=0_4': 142,
    '3DBall?team=0_5': 142,
    '3DBall?team=0_6': 141,
    '3DBall?team=0_7': 139,
    '3DBall?team=0_8': 139,
    '3DBall?team=0_9': 141},
    },

Ideally, get_steps(...) should the steps for all the agents in either the deicision_steps or terminal_steps.
However, occasionally, it may only return steps of some agents.

pprint(decision_steps)
<mlagents_envs.base_env.DecisionSteps object at 0x000001F5AB893370>
pprint(decision_steps.obs)
[array([], shape=(0, 8), dtype=float32)]
pprint(terminal_steps)
<mlagents_envs.base_env.TerminalSteps object at 0x000001F5AB8934C0>
pprint(terminal_steps.obs)
[array([[-0.19892487,  0.04448491,  3.0214067 ,  2.460743  , -0.8872237 ,
         2.8523436 , -2.9271135 , -0.36632335]], dtype=float32)]

These exceptions occur if Unity doesn't return steps for all agents when our episode_timesteps reaches the episode_horizon.

ray/rllib/env/wrappers/unity3d_env.py

Lines 181 to 188 in 29e6029

	if self.episode_timesteps > self.episode_horizon:
	return (
	obs,
	rewards,
	terminateds,
	dict({"__all__": True}, **{agent_id: True for agent_id in all_agents}),
	infos,
	)

Versions / Dependencies

Ray 2.40 (main)

Reproduction script

To reproduce this issue, train using the 3DBall environment from ML-Agents. Lowering the episode_horizon parameter in the constructor of Unity3DEnv can increase the likelihood of encountering the problem.

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[simonsays1980]

@Ginurx Thanks for raising this issue. We are right now overhauling our server-client setup to transfer it to our new API stack. So, this bug here is related and probably quite helpful.