MONAI Auto3DSeg "Hello World" Example using my own data failure #6785

keesh0 · 2023-07-27T01:36:07Z

keesh0
Jul 27, 2023

Hello,
I am a MONAI newbie.
I minimally modified https://github.com/Project-MONAI/tutorials/blob/main/auto3dseg/notebooks/auto3dseg_hello_world.ipynb
to use my own CT data.
a. I am wondering how to troubleshoot failures?
b. Is there a good log file to check?
c. How do I control running on the CPU vs GPU?
d. Why in the example do we need labels for test data?

sim_datalist = {
    "testing": [
        {"image": "test_image_001.nii.gz", "label": "test_label_001.nii.gz"},
        {"image": "test_image_002.nii.gz", "label": "test_label_002.nii.gz"},
    ],

569     retcode = process.poll()
   570     if check and retcode:
--> 571         raise CalledProcessError(retcode, process.args,
   572                                  output=stdout, stderr=stderr)
   573 return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['python', 'Y:\\data\\GMI\\3d_chewy_bars\\CT_training_images\\workdir\\dints_0\\scripts\\train.py', 'run', "--config_file='Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/hyper_parameters.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/hyper_parameters_search.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/network.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/network_search.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_infer.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_train.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_validate.yaml'", '--training#num_epochs_per_validation=1', '--training#num_images_per_batch=1', '--training#num_epochs=2']' returned non-zero exit status 1.

thx e.-

KumoLiu · 2023-07-27T03:08:56Z

KumoLiu
Jul 27, 2023
Maintainer

Hi @mingxin-zheng, could you please help share some comments on this one?
Thanks in advance!

1 reply

keesh0 Jul 27, 2023
Author

Sure, Should I include my small Python script?

mingxin-zheng · 2023-07-27T03:22:53Z

mingxin-zheng
Jul 27, 2023
Collaborator

Hi @keesh0 , some quick answers to your questions.
(a). We are using the standard logging to print the info/debug info to the screen. The training/evaluation is called by subprocess. In a linux environment, the error in the subprocess should print on the screen, a few lines above CalledProcessError. Do you see that in your environment?
(b). If you have errors in using a custom dataset, the best way is to copy the templates from here, and assign the template_path_or_url with your local path to the algorithm_templates as this doc. Then you can have a debugging session into the templates. If you are using VSCode, remember to enable subprocess as this guide
(c). So far Auto3DSeg requires a GPU to meet the minimum requirement. Some components allow CPU (data analyzer, ensembler), but the training needs GPU.
(d). Testing labels are only for visualization purposes. It is not necessary in the pipeline.

I do not have an active Windows development environment for now, but will spend some time to get one and can circle back after that.

0 replies

keesh0 · 2023-07-28T02:03:39Z

keesh0
Jul 28, 2023
Author

Thanks, I will try your ideas tomorrow and let you know.

2 replies

keesh0 Jul 29, 2023
Author

I moved from a Conda Jupyter notebook to a Python venv under IntelliJ.

Even though pytorch seems happy prior to calling monai.

GPU test:
tensor([[0.3805, 0.9021, 0.6086],
        [0.9291, 0.5423, 0.9038],
        [0.5019, 0.8194, 0.6001],
        [0.0987, 0.4764, 0.6473],
        [0.2432, 0.4825, 0.3983]])
Torch/Cuda is well: True, device: 1 NVIDIA GeForce RTX 3060
Num GPUs: 1
Has CUDA: True
CUDA version: 11.8
cuDNN enabled: True
cuDNN version: 8700
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90', 'compute_37']
GPU 0 Name: NVIDIA GeForce RTX 3060
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 28
GPU 0 Total memory (GB): 12.0
GPU 0 CUDA capability (maj.min): 8.6

Monai seems unhappy with trying to load pytorch?

2023-07-29 17:44:17,175 - INFO - AutoRunner using work directory Y:\data\GMI\3d_chewy_bars\CT_training_images\workdir
2023-07-29 17:44:17,176 - INFO - Setting num_fold 3 based on the input datalist Y:\data\GMI\3d_chewy_bars\CT_training_images\workdir\sim_datalist.json.
2023-07-29 17:44:17,318 - INFO - Using user defined command running prefix , will override other settings
2023-07-29 17:44:17,319 - INFO - Skipping data analysis...
2023-07-29 17:44:17,319 - INFO - Skipping algorithm generation...
2023-07-29 17:44:17,434 - INFO - The keys num_warmup_epochs cannot be found in the Y:\data\GMI\3d_chewy_bars\CT_training_images\workdir\dints_0\configs\hyper_parameters.yaml for training. Skipped overriding key num_warmup_epochs.
2023-07-29 17:44:17,435 - INFO - ['python', 'Y:\\data\\GMI\\3d_chewy_bars\\CT_training_images\\workdir\\dints_0\\scripts\\train.py', 'run', "--config_file='Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/hyper_parameters.yaml,Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/hyper_parameters_search.yaml,Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/network.yaml,Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/network_search.yaml,Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_infer.yaml,Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_train.yaml,Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_validate.yaml'", '--training#num_epochs_per_validation=1', '--training#num_images_per_batch=1', '--training#num_epochs=2']
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'torch'
Traceback (most recent call last):
  File "D:\source\keesh_programming\GMI\gmi-oatbars\p310__monai_venv\lib\site-packages\monai\utils\misc.py", line 850, in run_cmd
    return subprocess.run(cmd_list, **kwargs)
  File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', 'Y:\\data\\GMI\\3d_chewy_bars\\CT_training_images\\workdir\\dints_0\\scripts\\train.py', 'run', "--config_file='Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/hyper_parameters.yaml,Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/hyper_parameters_search.yaml,Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/network.yaml,Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/network_search.yaml,Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_infer.yaml,Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_train.yaml,Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_validate.yaml'", '--training#num_epochs_per_validation=1', '--training#num_images_per_batch=1', '--training#num_epochs=2']' returned non-zero exit status 1.

KumoLiu Jul 31, 2023
Maintainer

Hi @keesh0, seems you didn't install PyTorch. Could you please try to install it in your new env?
Thanks!

keesh0 · 2023-07-31T13:28:48Z

keesh0
Jul 31, 2023
Author

From pip list:
torch 2.0.1+cu118
torchvision 0.14.1

1 reply

wyli Jul 31, 2023
Collaborator

according to the log it seems there's system path issue, the Python libs are from different locations:

File "D:\source\keesh_programming\GMI\gmi-oatbars\p310__monai_venv\lib\site-packages\monai\utils\misc.py"
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\subprocess.py"

keesh0 · 2023-07-31T14:21:17Z

keesh0
Jul 31, 2023
Author

Thanks I will take a look.

1 reply

keesh0 Jul 31, 2023
Author

Same problem under Visual Studio 2022 using a clean venv.
Same problem under Conda using a clean venv.

Message=Command '['python', 'Y:\data\GMI\3d_chewy_bars\CT_training_images\workdir\dints_0\scripts\train.py', 'run', "--config_file='Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/hyper_parameters.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/hyper_parameters_search.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/network.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/network_search.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_infer.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_train.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_validate.yaml'", '--training#num_epochs_per_validation=1', '--training#num_images_per_batch=1', '--training#num_epochs=2']' returned non-zero exit status 1.
Source=Y:\data\GMI\3d_chewy_bars\CT_training_images\workdir\algorithm_templates\dints\scripts\algo.py
StackTrace:
File "Y:\data\GMI\3d_chewy_bars\CT_training_images\workdir\algorithm_templates\dints\scripts\algo.py", line 490, in train (Current frame)
return self._run_cmd(cmd, devices_info)
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\Monai.py", line 66, in
runner.run()
subprocess.CalledProcessError: Command '['python',
'Y:\data\GMI\3d_chewy_bars\CT_training_images\workdir\dints_0\scripts\train.py', 'run', "--config_file='Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/hyper_parameters.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/hyper_parameters_search.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/network.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/network_search.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_infer.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_train.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_validate.yaml'", '--training#num_epochs_per_validation=1', '--training#num_images_per_batch=1', '--training#num_epochs=2']' returned non-zero exit status 1.

This shows that Torch is fine.
Monai has trouble loading Torch under 3 clean environments,
GPU test:
tensor([[0.4117, 0.2132, 0.4343],
[0.2616, 0.4455, 0.4974],
[0.9297, 0.4850, 0.2622],
[0.3811, 0.9646, 0.1499],
[0.5074, 0.0497, 0.7135]])
Torch/Cuda is well: True, device: 1 NVIDIA GeForce RTX 3060
Num GPUs: 1
Has CUDA: True
CUDA version: 11.8
cuDNN enabled: True
cuDNN version: 8700
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90', 'compute_37']
GPU 0 Name: NVIDIA GeForce RTX 3060
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 28
GPU 0 Total memory (GB): 12.0
GPU 0 CUDA capability (maj.min): 8.6
2023-07-31 17:31:42,934 - INFO - AutoRunner using work directory Y:\data\GMI\3d_chewy_bars\CT_training_images\workdir
2023-07-31 17:31:42,936 - INFO - Setting num_fold 3 based on the input datalist Y:\data\GMI\3d_chewy_bars\CT_training_images\workdir\sim_datalist.json.
2023-07-31 17:31:43,064 - INFO - Skipping data analysis...
2023-07-31 17:31:43,064 - INFO - Skipping algorithm generation...
2023-07-31 17:31:43,166 - INFO - The keys num_warmup_epochs cannot be found in the Y:\data\GMI\3d_chewy_bars\CT_training_images\workdir\dints_0\configs\hyper_parameters.yaml for training. Skipped overriding key num_warmup_epochs.
2023-07-31 17:31:43,167 - INFO - Launching: python Y:\data\GMI\3d_chewy_bars\CT_training_images\workdir\dints_0\scripts\train.py run --config_file='Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/hyper_parameters.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/hyper_parameters_search.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/network.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/network_search.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_infer.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_train.yaml','Y:/data/GMI/3d_chewy_bars/CT_training_images/workdir/dints_0/configs/transforms_validate.yaml' --training#num_epochs_per_validation=1 --training#num_images_per_batch=1 --training#num_epochs=2
Traceback (most recent call last):
File "Y:\data\GMI\3d_chewy_bars\CT_training_images\workdir\dints_0\scripts\train.py", line 29, in
import torch
ModuleNotFoundError: No module named 'torch'
Backend TkAgg is interactive backend. Turning interactive mode on.

keesh0 · 2023-08-02T20:33:00Z

keesh0
Aug 2, 2023
Author

Am I missing something?

PS D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\monai_env> pip list
Package            Version
------------------ -----------
certifi            2023.7.22
charset-normalizer 3.2.0
contourpy          1.1.0
cycler             0.11.0
filelock           3.12.2
fire               0.5.0
fonttools          4.41.1
idna               3.4
Jinja2             3.1.2
kiwisolver         1.4.4
MarkupSafe         2.1.3
matplotlib         3.7.2
monai-weekly       1.3.dev2331
mpmath             1.3.0
networkx           3.1
nibabel            5.1.0
numpy              1.25.2
packaging          23.1
Pillow             10.0.0
pip                22.0.4
pyparsing          3.0.9
python-dateutil    2.8.2
PyYAML             6.0.1
requests           2.31.0
setuptools         58.1.0
six                1.16.0
sympy              1.12
termcolor          2.3.0
torch              2.0.1+cu118
torchvision        0.15.2
typing_extensions  4.7.1
urllib3            2.0.4

0 replies

keesh0 · 2023-08-02T22:42:41Z

keesh0
Aug 2, 2023
Author

D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\monai_env\Scripts\python.exe
2023-08-02 17:39:41,144 - INFO - AutoRunner using work directory Y:\data\GMI\3d_chewy_bars\CT_training_images\workdir
2023-08-02 17:39:41,145 - INFO - Setting num_fold 3 based on the input datalist Y:\data\GMI\3d_chewy_bars\CT_training_images\workdir\sim_datalist.json.
2023-08-02 17:39:41,147 - INFO - Running data analysis...
2023-08-02 17:39:41,147 - INFO - Found 1 GPUs for data analyzing!
0%| | 0/4 [00:00<?, ?it/s]
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\Monai.py", line 69, in
runner.run()
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\monai_env\lib\site-packages\monai\apps\auto3dseg\auto_runner.py", line 741, in run
da.get_all_case_stats()
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\monai_env\lib\site-packages\monai\apps\auto3dseg\data_analyzer.py", line 227, in get_all_case_stats
result_bycase = self._get_all_case_stats(0, 1, None, key, transform_list)
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\monai_env\lib\site-packages\monai\apps\auto3dseg\data_analyzer.py", line 326, in _get_all_case_stats
for batch_data in tqdm(dataloader) if (has_tqdm and rank == 0) else dataloader:
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\monai_env\lib\site-packages\tqdm\std.py", line 1178, in iter
for obj in iterable:
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\monai_env\lib\site-packages\torch\utils\data\dataloader.py", line 441, in iter
return self._get_iterator()
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\monai_env\lib\site-packages\torch\utils\data\dataloader.py", line 388, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\monai_env\lib\site-packages\torch\utils\data\dataloader.py", line 1042, in init
w.start()
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 45, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

0%| | 0/4 [00:05<?, ?it/s]
Backend TkAgg is interactive backend. Turning interactive mode on.

0 replies

KumoLiu · 2023-08-03T03:15:47Z

KumoLiu
Aug 3, 2023
Maintainer

Hi @keesh0, to keep things simple, you can run it without changing the "hello world" tutorial, right? If that's the case, it means your env is right. Then all you need to change is the "Generate images and labels" section, specifically the generate "datalist_file.json" file, correct?
So ensure that you are in the same env when running the original tutorial and only change "Generate images and labels" in this section.

0 replies

mingxin-zheng · 2023-08-03T13:23:29Z

mingxin-zheng
Aug 3, 2023
Collaborator

Hi @keesh0 @KumoLiu @wyli , sorry for my delayed response. I managed to get a Windows machine here and replicated the bug.

CalledProcessError: Command '['python', 'c:\\Users\\Admininstrator\\tutorials\\auto3dseg\\notebooks\\helloworld_work_dir\\dints_0\\scripts\\train.py', 'run', "--config_file='c:/Users/Admininstrator/tutorials/auto3dseg/notebooks/helloworld_work_dir/dints_0/configs/hyper_parameters.yaml','c:/Users/Admininstrator/tutorials/auto3dseg/notebooks/helloworld_work_dir/dints_0/configs/hyper_parameters_search.yaml','c:/Users/Admininstrator/tutorials/auto3dseg/notebooks/helloworld_work_dir/dints_0/configs/network.yaml','c:/Users/Admininstrator/tutorials/auto3dseg/notebooks/helloworld_work_dir/dints_0/configs/network_search.yaml','c:/Users/Admininstrator/tutorials/auto3dseg/notebooks/helloworld_work_dir/dints_0/configs/transforms_infer.yaml','c:/Users/Admininstrator/tutorials/auto3dseg/notebooks/helloworld_work_dir/dints_0/configs/transforms_train.yaml','c:/Users/Admininstrator/tutorials/auto3dseg/notebooks/helloworld_work_dir/dints_0/configs/transforms_validate.yaml'", '--training#num_epochs_per_validation=1', '--training#num_images_per_batch=2', '--training#num_epochs=2']' returned non-zero exit status 1.

The cause of this is:

Traceback (most recent call last):
  File "c:\Users\Admininstrator\tutorials\auto3dseg\notebooks\helloworld_work_dir\dints_0\scripts\train.py", line 54, in <module>
    _libcudart = ctypes.CDLL("libcudart.so")
  File "C:\Users\Admininstrator\miniconda3\envs\py38\lib\ctypes\__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'libcudart.so' (or one of its dependencies). Try using the full path with constructor syntax.

The try ... catch... here did not handle the FileNotFoundError on Windows platform.
https://github.com/Project-MONAI/research-contributions/blob/7306d8d3561148a19b3078887ff520b7ceb9e8a0/auto3dseg/algorithm_templates/dints/scripts/train.py#L56

2 replies

mingxin-zheng Aug 4, 2023
Collaborator

@KumoLiu can we convert this as an issue? I try to catch the FileNotFoundError in the dints template in the research-contribution but got another issue:

python c:\\Users\\Administrator\\tutorials\\auto3dseg\\notebooks\\helloworld_work_dir\\dints_0\\scripts\\train.py run "--config_file='c:/Users/Administrator/tutorials/auto3dseg/notebooks/helloworld_work_dir/dints_0/configs/hyper_parameters.yaml','c:/Users/Administrator/tutorials/auto3dseg/notebooks/helloworld_work_dir/dints_0/configs/hyper_parameters_search.yaml','c:/Users/Administrator/tutorials/auto3dseg/notebooks/helloworld_work_dir/dints_0/configs/network.yaml','c:/Users/Administrator/tutorials/auto3dseg/notebooks/helloworld_work_dir/dints_0/configs/network_search.yaml','c:/Users/Administrator/tutorials/auto3dseg/notebooks/helloworld_work_dir/dints_0/configs/transforms_infer.yaml','c:/Users/Administrator/tutorials/auto3dseg/notebooks/helloworld_work_dir/dints_0/configs/transforms_train.yaml','c:/Users/Administrator/tutorials/auto3dseg/notebooks/helloworld_work_dir/dints_0/configs/transforms_validate.yaml'" --training#num_epochs_per_validation=1 --training#num_images_per_batch=2 --training#num_epochs=2
Warning: cannot find libcudart.so or AMD ROCm platform, set device limit is disabled
dints_0 - training ...:   0%|                                                                                     | 0/2 [00:00<?, ?round/s]Warning: cannot find libcudart.so or AMD ROCm platform, set device limit is disabled
Warning: cannot find libcudart.so or AMD ROCm platform, set device limit is disabled
Warning: cannot find libcudart.so or AMD ROCm platform, set device limit is disabled
Warning: cannot find libcudart.so or AMD ROCm platform, set device limit is disabled
Warning: cannot find libcudart.so or AMD ROCm platform, set device limit is disabled
Warning: cannot find libcudart.so or AMD ROCm platform, set device limit is disabled
Warning: cannot find libcudart.so or AMD ROCm platform, set device limit is disabled
Warning: cannot find libcudart.so or AMD ROCm platform, set device limit is disabled
dints_0 - training ...:   0%|                                                                                     | 0/2 [01:13<?, ?round/s]
Traceback (most recent call last):
  File "c:\Users\Administrator\tutorials\auto3dseg\notebooks\helloworld_work_dir\dints_0\scripts\train.py", line 934, in <module>
    fire.Fire()
  File "C:\Users\Administrator\miniconda3\envs\py38\lib\site-packages\fire\core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "C:\Users\Administrator\miniconda3\envs\py38\lib\site-packages\fire\core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "C:\Users\Administrator\miniconda3\envs\py38\lib\site-packages\fire\core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "c:\Users\Administrator\tutorials\auto3dseg\notebooks\helloworld_work_dir\dints_0\scripts\train.py", line 556, in run
    scaler.scale(loss).backward()
  File "C:\Users\Administrator\miniconda3\envs\py38\lib\site-packages\torch\_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "C:\Users\Administrator\miniconda3\envs\py38\lib\site-packages\torch\autograd\__init__.py", line 200, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

KumoLiu Aug 4, 2023
Maintainer

Sure.

mingxin-zheng · 2023-08-04T04:08:31Z

mingxin-zheng
Aug 4, 2023
Collaborator

Hi @keesh0 ,

Thanks for opening the issue. I think this is a bug in the Auto3DSeg DiNTs and SwinUNETR template, and we need to address it.

Now the "segresnet" (for 3D) and "segrenset2d" (for 2.5D images that has sparsely-sampled images on the z-axis) should work. I pick the "segresnet" for the hello-world example as below and it runs fine on the Windows platform

runner = AutoRunner(
    work_dir=work_dir,
    input={
        "modality": "MRI",
        "datalist": datalist_file,
        "dataroot": dataroot_dir,
    },
    algos = "segresnet"
)

1 reply

dzenanz Nov 1, 2023

Adding algos = "segresnet" does not seem to help me. My config is here: #6821 (comment)

keesh0 · 2023-08-05T23:07:18Z

keesh0
Aug 5, 2023
Author

Should I update my version of Monai?
Do you recommend trying Monai on a different OS like MAC or Linux?
thx

0 replies

keesh0 · 2023-08-08T20:57:27Z

keesh0
Aug 8, 2023
Author

I tried the Hello World simulated data example and got the following error:
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

2023-08-08 15:54:37,360 - INFO - AutoRunner using work directory ./helloworld_work_dir
2023-08-08 15:54:37,362 - INFO - Setting num_fold 3 based on the input datalist D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\helloworld_work_dir\sim_datalist.json.
2023-08-08 15:54:37,372 - INFO - Running data analysis...
2023-08-08 15:54:37,372 - INFO - Found 1 GPUs for data analyzing!

0%| | 0/12 [00:00<?, ?it/s]
0%| | 0/12 [00:00<?, ?it/s]
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\Monai_synth.py", line 93, in
runner.run()
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\monai_env\lib\site-packages\monai\apps\auto3dseg\auto_runner.py", line 741, in run
da.get_all_case_stats()
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\monai_env\lib\site-packages\monai\apps\auto3dseg\data_analyzer.py", line 227, in get_all_case_stats
result_bycase = self._get_all_case_stats(0, 1, None, key, transform_list)
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\monai_env\lib\site-packages\monai\apps\auto3dseg\data_analyzer.py", line 326, in _get_all_case_stats
for batch_data in tqdm(dataloader) if (has_tqdm and rank == 0) else dataloader:
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\monai_env\lib\site-packages\tqdm\std.py", line 1178, in iter
for obj in iterable:
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\monai_env\lib\site-packages\torch\utils\data\dataloader.py", line 441, in iter
return self._get_iterator()
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\monai_env\lib\site-packages\torch\utils\data\dataloader.py", line 388, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\monai_env\lib\site-packages\torch\utils\data\dataloader.py", line 1042, in init
w.start()
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 45, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\keesh\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

0%| | 0/12 [00:10<?, ?it/s]
DataLoader worker (pid(s) 110248, 79900, 85568, 81392) exited unexpectedly
Stack trace:

_queue.Empty:

The above exception was the direct cause of the following exception:

File "D:\source\keesh_programming\GMI\monai_viz_studio\Monai\Monai\Monai_synth.py", line 93, in (Current frame)
runner.run()
RuntimeError: DataLoader worker (pid(s) 110248, 79900, 85568, 81392) exited unexpectedly
Loaded 'torch.utils.data.dataloader'
Loaded 'tqdm.std'
Loaded 'monai.apps.auto3dseg.data_analyzer'
Loaded 'monai.apps.auto3dseg.auto_runner'
Loaded 'queue'

0 replies

mingxin-zheng · 2023-08-09T02:20:02Z

mingxin-zheng
Aug 9, 2023
Collaborator

I met this error when I tried to debug something, specifically, when I use VSCode to debug python code. Do you see this in a debugger as well @keesh0 ?

3 replies

keesh0 Aug 9, 2023
Author

Yes, debugging under Visual Studio 2022.
Let me try running under Power Shell.
...
Same error outside the debugger under PowerShell.
Is it Monai or auto3dseg itself which is unstable under Windows?

Itsanewday Apr 8, 2024

Did you solve this problem? How can i train a segresnet model under Windows?

dzenanz Apr 8, 2024

Auto3DSeg does not work on Windows, as per #6821 (comment).

keesh0 · 2023-08-16T21:42:27Z

keesh0
Aug 16, 2023
Author

Is there a new version of monai.apps.auto3dseg that would fix my issues under Windows 11?
Is there another automatic 3D classification system under Monai which you would recommend?

0 replies

keesh0 · 2024-04-09T14:30:16Z

keesh0
Apr 9, 2024
Author

I had to move onto another framework, mmsegmentation.
But, I may explore your GAN support / examples.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MONAI Auto3DSeg "Hello World" Example using my own data failure #6785

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 15 comments 11 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

MONAI Auto3DSeg "Hello World" Example using my own data failure #6785

keesh0 Jul 27, 2023

Replies: 15 comments · 11 replies

KumoLiu Jul 27, 2023 Maintainer

keesh0 Jul 27, 2023 Author

mingxin-zheng Jul 27, 2023 Collaborator

keesh0 Jul 28, 2023 Author

keesh0 Jul 29, 2023 Author

KumoLiu Jul 31, 2023 Maintainer

keesh0 Jul 31, 2023 Author

wyli Jul 31, 2023 Collaborator

keesh0 Jul 31, 2023 Author

keesh0 Jul 31, 2023 Author

keesh0 Aug 2, 2023 Author

keesh0 Aug 2, 2023 Author

KumoLiu Aug 3, 2023 Maintainer

mingxin-zheng Aug 3, 2023 Collaborator

mingxin-zheng Aug 4, 2023 Collaborator

KumoLiu Aug 4, 2023 Maintainer

mingxin-zheng Aug 4, 2023 Collaborator

dzenanz Nov 1, 2023

keesh0 Aug 5, 2023 Author

keesh0 Aug 8, 2023 Author

mingxin-zheng Aug 9, 2023 Collaborator

keesh0 Aug 9, 2023 Author

Itsanewday Apr 8, 2024

dzenanz Apr 8, 2024

keesh0 Aug 16, 2023 Author

keesh0 Apr 9, 2024 Author

keesh0
Jul 27, 2023

Replies: 15 comments 11 replies

KumoLiu
Jul 27, 2023
Maintainer

keesh0 Jul 27, 2023
Author

mingxin-zheng
Jul 27, 2023
Collaborator

keesh0
Jul 28, 2023
Author

keesh0 Jul 29, 2023
Author

KumoLiu Jul 31, 2023
Maintainer

keesh0
Jul 31, 2023
Author

wyli Jul 31, 2023
Collaborator

keesh0
Jul 31, 2023
Author

keesh0 Jul 31, 2023
Author

keesh0
Aug 2, 2023
Author

keesh0
Aug 2, 2023
Author

KumoLiu
Aug 3, 2023
Maintainer

mingxin-zheng
Aug 3, 2023
Collaborator

mingxin-zheng Aug 4, 2023
Collaborator

KumoLiu Aug 4, 2023
Maintainer

mingxin-zheng
Aug 4, 2023
Collaborator

keesh0
Aug 5, 2023
Author

keesh0
Aug 8, 2023
Author

mingxin-zheng
Aug 9, 2023
Collaborator

keesh0 Aug 9, 2023
Author

keesh0
Aug 16, 2023
Author

keesh0
Apr 9, 2024
Author