I have a stress test where many pyxis containers are ran in sequence on a node with the same container filesystem (--container-name).
Pyxis calls enroot list -f to check if a container exists and if it's already running.
But it seems that enroot list -f can fail with return code 1 if a running container exits during the invocation of enroot list -f.

It seems to be caused by the ps -p command:

In rare occasions, I also saw a return code of 2 and it's likely caused by a similar race between the lsns and this call:

Example where it happens (might need a few attempts and/or tweaking the sleep duration):

$ enroot import -o /tmp/pytorch.sqsh docker://nvcr.io#nvidia/pytorch:22.05-py3
$ enroot create --force --name pyxis_pytorch /tmp/pytorch.sqsh
$ { enroot start --root --rw pyxis_pytorch sleep 1s &>/dev/null & }; sleep 1.4 ; enroot list -f; echo $?; wait; 
[1] 1536191
NAME  PID  COMM  STATE  STARTED  TIME  MNTNS  USERNS  COMMAND
[1]+  Done                    enroot start --root --rw pyxis_pytorch sleep 1s &> /dev/null
1