Memory Leak in GPU During Continuous Scene Loading and Coverage Map Calculation #662
Comments
|
I encountered this issue as well. In a fixed scene, I have about 10 transmitters, and each transmitter corresponds to approximately 1500 receiver positions. During the simulation, I simulate only one transmitter at a time, and for each simulation, I process a batch of 120 receivers at a time. After the calculation, I remove the receivers and the computed paths from the scene. Usually, there is no issue when calculating the first transmitter, but after the calculation of the second transmitter, I get an out-of-resources error. I suspect that something in the scene wasn't properly cleared, which is causing this. batch_size = 120
for idx, (x_tx, y_tx, z_tx) in tqdm(enumerate(tx_position), total=len(tx_position), desc="Simulating Transmitter Positions", unit="transmitter"):
tx = Transmitter(name=f"tx-{idx}",
position=tx_position[idx])
scene.add(tx)
rx_pos = rx_position[idx]
rx_tmp_len = len(rx_pos)
# 动态按批次处理接收机位置
receiver_batches = len(rx_pos) // batch_size + (1 if len(rx_pos) % batch_size != 0 else 0)
for batch_start in tqdm(range(0, len(rx_pos), batch_size),
total=receiver_batches,
desc=f"Processing Receivers for tx-{idx}",
unit="batch",
leave=False,
dynamic_ncols=True):
# 获取当前批次的接收机位置
rx_batch = rx_pos[batch_start:batch_start + batch_size]
for i, rx in enumerate(rx_batch):
receiver_name = f"rx-{batch_start + i}"
rx = Receiver(name=receiver_name,
position=rx)
scene.add(rx)
# 计算路径
paths = scene.compute_paths(max_depth=5,
num_samples=1e6,
diffraction=True,
scattering=True)
# 移除当前批次的接收机
for i in range(len(rx_batch)):
scene.remove(f"rx-{batch_start + i}")
del paths # Free memory
scene.remove(f"tx-{idx}")
Among all the resources, my GPU memory is fully utilized. I have already set the following: if gpus:
try:
tf.config.experimental.set_memory_growth(gpus[0], True)
except RuntimeError as e:
print(e)
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm encountering a GPU memory leak while performing repeated scene loading and coverage map calculations for my research work. The issue occurs specifically when executing new scene and the
coverage_mapfunction within a loop. Interestingly, thecompute_pathsfunction doesn't exhibit this problem under similar conditions.The image shows the linear increase in memory when I load new scene and perform
coverage_mapfunction.I suspect that after loading a scene and computing the coverage map, some global parameters or cached data might be persisting in the GPU memory, leading to the observed leak.
Could you please investigate this issue? Specifically, I'd appreciate if you could:
Any insights or solutions to resolve this memory leak would be greatly appreciated. Thank you for your time and assistance.
My system:
The text was updated successfully, but these errors were encountered: