How to control parameters of Auto3DSeg to allow running on lower-resource computer?

[dzenanz]

Due to Auto3DSeg not being able to run on Windows, I am trying to run it on my secondary machine with Ubuntu 22.04. However it is a lot older, with 4-core CPU and 2GB GPU. Trying to run the hello world runs into:

RuntimeError: subprocess call error 1: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.

and later

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacty of 1.95 GiB of which 2.12 MiB is free. Process 14434 has 638.12 MiB memory in use. Process 14433 has 24.00 MiB memory in use. Process 71412 has 2.98 MiB memory in use. Including non-PyTorch memory, this process has 1.12 GiB memory in use. Of the allocated memory 1.07 GiB is allocated by PyTorch, and 15.26 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Is there a way to control parallelism in Auto3DSeg?

[KumoLiu]

Hi @dzenanz, you could refer to "Override the training parameters so that we can complete the pipeline in minutes" about how to set training parameters (such as num_workers) in this tutorial.
BTW, you could set device information by using set_device_info.

MONAI/monai/apps/auto3dseg/auto_runner.py

Line 512 in 782c1e6

def set_device_info(

Hope it helps, thanks!

[dzenanz]

I already had the 2-epoch settings. I set the number of workers to 1 as per your suggestion, that fixes the warning. I tried reducing the simulated image size to 16x16x16. I still run out of memory. It is unexpected that a 2GB GPU chokes on 16^3 images. I pushed my changes and my run results to my fork:
https://github.com/dzenanz/MONAI-tutorials/blob/main/auto3dseg/notebooks/auto3dseg_hello_world.ipynb

Please advise further.

[KumoLiu]

Hi @dzenanz, perhaps you need to set roi_size in the train_param as well.
https://github.com/Project-MONAI/research-contributions/blob/a8c9939e9d4e1448411d41b5a89e06b22d693015/auto3dseg/algorithm_templates/segresnet/configs/hyper_parameters.yaml#L19
You could also check the data stats.
data stats: media/xeonatorM/Dev/MONAI-tutorials/auto3dseg/notebooks/helloworld_work_dir/datastats.yaml
by-case data stats: /media/xeonatorM/Dev/MONAI-tutorials/auto3dseg/notebooks/helloworld_work_dir/datastats_by_case.yaml
Thanks!

[dzenanz]

I tried setting roi_size, and realized that dints ignores it. Then I tried requesting segresnet, and that ran to completion. I pushed changes to my fork. Now it is time to slowly expand this, and see how far I get with this computer.

[dzenanz]

Returning image dimensions to 64^3 makes it run out of memory again. I now have a new question: #7221.

[marksgraham]

Relatedly - are the full set of options for training_params documented anywhere? In my case I'm trying to prevent my data being cached during training, but more generally I'd like to know how to figure out how I can adjust other options.

CC @pwrightkcl

[KumoLiu]

Hi @marksgraham, we don't have comprehensive documentation about what params users can override, would be great to add one.
For now, perhaps the easiest way maybe check the algorithm templates. The problem that may arise is that different templates do not necessarily use the same variables. We are also trying to unify them.
https://github.com/Project-MONAI/research-contributions/blob/main/auto3dseg/algorithm_templates/segresnet/configs/hyper_parameters.yaml

For the cached issue you raised, you could specify cache_rate.
Thanks!