GPU Memory release

[Mariusmarten]

I observed the following problem: Carla does not release the GPU 'freed' by destroying sensors.

CARLA version: 0.9.14

Platform/OS: carlasim/carla:0.9.14 docker image

Problem: GPU memory usage keeps on increasing. Once spawned sensors continue to occupy GPU space after being destroyed, the space is partially freed up before the GPU is almost full but this leads still to the data generation crashing at some point. Just to be clear the number of actors (vehicles and cameras) stays constant they are just spawned and destroyed multiple times.

The error I get is:

Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554
CommonUnixCrashHandler: Signal=11
Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
Segmentation fault

Expected: The GPU is directly freed up when the sensor is destroyed.

Steps to reproduce: Create and destroy sensors multiple times (e.g., spawn 24 camera sensors and destroy them 100 times in a loop).

Other information: I could not find further information about others experiencing similar problems or whether this is potentially expected behavior.

Potentially related to: this, this, this and this

[Mariusmarten]

Hi, I am on a Linux-based server system in a docker build on carlasim/carla:0.9.14 using this whl carla-0.9.14-cp38-cp38-manylinux_2_27_x86_64.whl. I have no problem running Carla otherwise. For me, the crash is very deterministically due to the not-released GPU memory when sensors are deleted and respawnd.

[Blyron]

This should be fixed in latest versions of ue4-dev branch