GPU requirements to support inference with pretrained segmentation model.

[bkonk]

I'm currently running 3D slicer in a Windows ec2 environment, with a monailabel server running in a docker container on a p3.8xlarge ec2 instance. I'm using the pretrained "segmentation" model, and hitting CUDA out of memory errors when selecting "Next Sample" to bring up my first sample. The p3.8xlarge has 4xGPUs, each with 16 GiB, but it sounds like multi-GPU doesn't support inference and "Next Sample" is attempting inference... so I'm essentially running with 16GiB anyway. This is cardiac imaging, so high slice number and 512x512. I'm assuming this might work with 16GiB if I had smaller volumes?

Just wondering if there are specific GPU memory requirements for AIAA. Is this behavior expected when running on a gpu of this size? Is there any workaround?

[SachidanandAlle]

Please share logs from the server..
You can run inference on a specific GPU.. if you are using 3D Slicer, you can find an option to disable running inference on Next sample.. That way you can control to run it manually over specific GPU..

this is more to understand the exact problem.. is it GPU... is it 16 GB memory.. is it due to image size..
If you are having small GPU memory to process larger volumes.. some post processing can happen on virtual memory (or CPU) instead of GPU..

you get the best throughput when you have large GPU memory to fit your input image. If not, then you switch some of the pre-processing transforms to run on CPU instead of GPU

[SachidanandAlle]

For example, you can remove this transform to run in CPU instead of GPU
https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/infers/segmentation.py#L65

i don't know which model you are running.. so if you can provide more the details and logs, others can help you to debug

[bkonk]

I'm using the default segmentation model in the radiology app.

Using PYTHONPATH=/usr:

[2023-09-05 17:35:31,828] [542] [MainThread] [INFO] (main:285) - USING:: version = False
[2023-09-05 17:35:31,828] [542] [MainThread] [INFO] (main:285) - USING:: app = /opt/monai/apps/radiology
[2023-09-05 17:35:31,828] [542] [MainThread] [INFO] (main:285) - USING:: studies = /data
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: verbose = INFO
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: conf = [['models', 'segmentation']]
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: host = 0.0.0.0
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: port = 8000
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: uvicorn_app = monailabel.app:app
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: ssl_keyfile = None
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: ssl_certfile = None
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: ssl_keyfile_password = None
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: ssl_ca_certs = None
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: workers = None
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: limit_concurrency = None
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: access_log = False
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: root_path = /
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: log_level = info
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: log_config = None
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: dryrun = False
[2023-09-05 17:35:31,829] [542] [MainThread] [INFO] (main:285) - USING:: action = start_server
[2023-09-05 17:35:31,830] [542] [MainThread] [INFO] (main:296) -
[2023-09-05 17:35:32,316] [542] [MainThread] [INFO] (uvicorn.error:74) - Started server process [542]
[2023-09-05 17:35:32,317] [542] [MainThread] [INFO] (uvicorn.error:48) - Waiting for application startup.
[2023-09-05 17:35:32,317] [542] [MainThread] [INFO] (monailabel.interfaces.utils.app:37) - Initializing App from: /opt/monai/apps/radiology; studies: /data; conf: {'models': 'segmentation'}
[2023-09-05 17:35:33,060] [542] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2023-09-05 17:35:33,066] [542] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_vertebra.LocalizationVertebra'>
[2023-09-05 17:35:33,068] [542] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2023-09-05 17:35:33,068] [542] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_spine.LocalizationSpine'>
[2023-09-05 17:35:33,069] [542] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2023-09-05 17:35:33,069] [542] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_vertebra.SegmentationVertebra'>
[2023-09-05 17:35:34,333] [542] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2023-09-05 17:35:34,333] [542] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2023-09-05 17:35:34,334] [542] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2023-09-05 17:35:34,334] [542] [MainThread] [INFO] (main:93) - +++ Adding Model: segmentation => lib.configs.segmentation.Segmentation
[2023-09-05 17:35:34,519] [542] [MainThread] [INFO] (main:96) - +++ Using Models: ['segmentation']
[2023-09-05 17:35:34,519] [542] [MainThread] [INFO] (monailabel.interfaces.app:135) - Init Datastore for: /data
[2023-09-05 17:35:34,520] [542] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: True; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '.xml']
[2023-09-05 17:35:34,523] [542] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2023-09-05 17:35:34,523] [542] [MainThread] [INFO] (monailabel.datastore.local:151) - Start observing external modifications on datastore (AUTO RELOAD)
[2023-09-05 17:35:34,545] [542] [MainThread] [INFO] (main:126) - +++ Adding Inferer:: segmentation => <lib.infers.segmentation.Segmentation object at 0x7f12d5fe0e50>
[2023-09-05 17:35:34,545] [542] [MainThread] [INFO] (main:191) - {'segmentation': <lib.infers.segmentation.Segmentation object at 0x7f12d5fe0e50>, 'Histogram+GraphCut': <monailabel.scribbles.infer.HistogramBasedGraphCut object at 0x7f12cc368f10>, 'GMM+GraphCut': <monailabel.scribbles.infer.GMMBasedGraphCut object at 0x7f12cc368f70>}
[2023-09-05 17:35:34,545] [542] [MainThread] [INFO] (main:206) - +++ Adding Trainer:: segmentation => <lib.trainers.segmentation.Segmentation object at 0x7f12cc368fd0>
[2023-09-05 17:35:34,545] [542] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /root/.cache/monailabel/sessions
[2023-09-05 17:35:34,546] [542] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2023-09-05 17:35:34,546] [542] [MainThread] [INFO] (monailabel.interfaces.app:469) - App Init - completed
Allow Origins: ['*']
[2023-09-05 17:35:34,546] [timeloop] [INFO] Starting Timeloop..
[2023-09-05 17:35:34,546] [542] [MainThread] [INFO] (timeloop:60) - Starting Timeloop..
[2023-09-05 17:35:34,546] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x7f12cf485280>
[2023-09-05 17:35:34,546] [542] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x7f12cf485280>
[2023-09-05 17:35:34,546] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
[2023-09-05 17:35:34,546] [542] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set
[2023-09-05 17:35:34,547] [542] [MainThread] [INFO] (uvicorn.error:62) - Application startup complete.
[2023-09-05 17:35:34,547] [542] [MainThread] [INFO] (uvicorn.error:217) - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[2023-09-05 17:36:23,337] [542] [MainThread] [INFO] (monailabel.endpoints.activelearning:44) - Active Learning Request: {'strategy': 'random', 'client_id': 'user-xyz'}
[2023-09-05 17:36:23,337] [542] [MainThread] [INFO] (monailabel.tasks.activelearning.random:47) - Random: Selected Image: 1107785143_HEART_NAV_40; Weight: 340073
[2023-09-05 17:36:23,339] [542] [MainThread] [INFO] (monailabel.endpoints.activelearning:60) - Next sample: {'id': '1107785143_HEART_NAV_40', 'weight': 340073, 'path': '/data/1107785143_HEART_NAV_40.nii.gz', 'ts': 1693411277, 'name': '1107785143_HEART_NAV_40.nii.gz', 'strategy': {'random': {'ts': 1693935383, 'client_id': 'user-xyz'}}}
[2023-09-05 17:36:24,942] [542] [MainThread] [INFO] (monailabel.endpoints.infer:161) - Infer Request: {'model': 'segmentation', 'image': '1107785143_HEART_NAV_40', 'largest_cc': True, 'device': 'Tesla V100-SXM2-16GB', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz'}
[2023-09-05 17:36:24,943] [542] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:277) - Infer Request (final): {'largest_cc': True, 'device': 'cuda', 'model': 'segmentation', 'image': '/data/1107785143_HEART_NAV_40.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'description': 'A pre-trained model for volumetric (3D) Segmentation from CT image'}
monai.transforms.io.dictionary LoadImaged.init:image_only: Current default value of argument image_only=False has been deprecated since version 1.1. It will be changed to image_only=True in version 1.3.
[2023-09-05 17:36:24,945] [542] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s)
[2023-09-05 17:36:24,945] [542] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['largest_cc', 'device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path']
[2023-09-05 17:36:27,005] [542] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (LoadImaged): Time: 2.059; image: torch.Size([512, 512, 256])(torch.float32)
[2023-09-05 17:36:27,125] [542] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureTyped): Time: 0.1201; image: torch.Size([512, 512, 256])(torch.float32)
[2023-09-05 17:36:27,126] [542] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureChannelFirstd): Time: 0.0003; image: torch.Size([1, 512, 512, 256])(torch.float32)
[2023-09-05 17:36:27,153] [542] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Orientationd): Time: 0.0272; image: torch.Size([1, 512, 512, 256])(torch.float32)
[2023-09-05 17:36:27,789] [542] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Spacingd): Time: 0.6353; image: torch.Size([1, 167, 167, 107])(torch.float32)
[2023-09-05 17:36:27,812] [542] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (NormalizeIntensityd): Time: 0.0225; image: torch.Size([1, 167, 167, 107])(torch.float32)
[2023-09-05 17:36:28,013] [542] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (GaussianSmoothd): Time: 0.2012; image: torch.Size([1, 167, 167, 107])(torch.float32)
[2023-09-05 17:36:28,030] [542] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityd): Time: 0.0159; image: torch.Size([1, 167, 167, 107])(torch.float32)
[2023-09-05 17:36:28,030] [542] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:490) - Inferer:: cuda => SlidingWindowInferer => {'roi_size': (96, 96, 96), 'sw_batch_size': 2, 'overlap': 0.4, 'mode': gaussian, 'sigma_scale': 0.125, 'padding_mode': 'replicate', 'cval': 0.0, 'sw_device': None, 'device': None, 'progress': False, 'cpu_thresh': None, 'buffer_steps': None, 'buffer_dim': -1, 'roi_weight_map': None}
[2023-09-05 17:36:28,030] [542] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:418) - Infer model path: /opt/monai/apps/radiology/model/pretrained_segmentation.pt
[2023-09-05 17:36:29,258] [542] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - INV - Run Transform(s)
[2023-09-05 17:36:29,258] [542] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - INV - Input Keys: ['largest_cc', 'device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'image_meta_dict', 'latencies', 'pred']
[2023-09-05 17:36:29,395] [542] [MainThread] [ERROR] (uvicorn.error:434) - Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/uvicorn/protocols/http/h11_impl.py", line 429, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.8/dist-packages/uvicorn/middleware/proxy_headers.py", line 78, in call
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.8/dist-packages/fastapi/applications.py", line 276, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.8/dist-packages/starlette/applications.py", line 122, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/cors.py", line 84, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/exceptions.py", line 79, in call
raise exc
File "/usr/local/lib/python3.8/dist-packages/starlette/middleware/exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "/usr/local/lib/python3.8/dist-packages/fastapi/middleware/asyncexitstack.py", line 21, in call
raise e
File "/usr/local/lib/python3.8/dist-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 718, in call
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.8/dist-packages/starlette/routing.py", line 66, in app
response = await func(request)
File "/usr/local/lib/python3.8/dist-packages/fastapi/routing.py", line 237, in app
raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.8/dist-packages/fastapi/routing.py", line 163, in run_endpoint_function
return await dependant.call(**values)
File "/usr/local/lib/python3.8/dist-packages/monailabel/endpoints/infer.py", line 180, in api_run_inference
return run_inference(background_tasks, model, image, session_id, params, file, label, output)
File "/usr/local/lib/python3.8/dist-packages/monailabel/endpoints/infer.py", line 162, in run_inference
result = instance.infer(request)
File "/usr/local/lib/python3.8/dist-packages/monailabel/interfaces/app.py", line 307, in infer
result_file_name, result_json = task(request)
File "/usr/local/lib/python3.8/dist-packages/monailabel/tasks/infer/basic_infer.py", line 311, in call
data = self.run_invert_transforms(data, pre_transforms, self.inverse_transforms(data))
File "/usr/local/lib/python3.8/dist-packages/monailabel/tasks/infer/basic_infer.py", line 406, in run_invert_transforms
d = run_transforms(d, transforms, inverse=True, log_prefix="INV")
File "/usr/local/lib/python3.8/dist-packages/monailabel/interfaces/utils/transform.py", line 95, in run_transforms
data = t.inverse(data)
File "/opt/monai/monai/transforms/spatial/dictionary.py", line 522, in inverse
d[key] = self.spacing_transform.inverse(cast(torch.Tensor, d[key]))
File "/opt/monai/monai/transforms/spatial/array.py", line 547, in inverse
return self.sp_resample.inverse(data)
File "/opt/monai/monai/transforms/spatial/array.py", line 248, in inverse
out: torch.Tensor = SpatialResample.call(self, data, **kw_args)
File "/opt/monai/monai/transforms/spatial/array.py", line 223, in call
return spatial_resample(
File "/opt/monai/monai/transforms/spatial/functional.py", line 178, in spatial_resample
img = affine_xform(img.unsqueeze(0), theta=xform.to(img), spatial_size=spatial_size).squeeze(0) # type: ignore
File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/monai/monai/networks/layers/spatial_transforms.py", line 580, in forward
dst = nn.functional.grid_sample(
File "/usr/local/lib/python3.8/dist-packages/torch/nn/functional.py", line 4244, in grid_sample
return torch.grid_sampler(input, grid, mode_enum, padding_mode_enum, align_corners)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 13.00 GiB (GPU 0; 15.77 GiB total capacity; 2.75 GiB already allocated; 12.27 GiB free; 3.12 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2023-09-05 17:37:04,079] [542] [MainThread] [INFO] (uvicorn.error:263) - Shutting down
[2023-09-05 17:37:04,180] [542] [MainThread] [INFO] (uvicorn.error:67) - Waiting for application shutdown.
[2023-09-05 17:37:04,180] [542] [MainThread] [INFO] (uvicorn.error:78) - Application shutdown complete.
[2023-09-05 17:37:04,180] [542] [MainThread] [INFO] (uvicorn.error:84) - Finished server process [542]

[diazandr3s]

Hi @bkonk,

Images are quite big 512, 512, 256 - How many regions are you trying to segment?

The OOM issue occurs when trying to invert the pre-transforms. Basically restoring spacing and orientation.

Could you please try commenting out these two lines and add this argument (invert_orient = True) to the Restored transform?

[SachidanandAlle]

Or just another solution to avoid GPU for pre transforms..

Comment EnsureTyped transform