High memory consumption for inference

[PushpakBhoge]

I have noticed that during inference memory consumption shoots up
take a look at this code

model = init_detector(**params)
gpu_memoy_idle = <something>

results = inference_detector(**params)
gpu_memory_after_inference = <something>

diff = gpu_memory_after_inference - gpu_memoy_idle
print(diff )

ideally, the diff should be approximately zero (a 10-30 MB difference is acceptable)
but I found out that this difference is coming out very high varying from 400 MB to 1500 MB, and this consumption goes up to a few inferences then it settles

why is this happening? This actually causes issues when the model is deployed using the Torchserve the memory usage shoots up