about the active learning #1419
Replies: 2 comments 3 replies
-
|
Hi @spencer03172023 , thanks for trying the app. From above logs, we didn't see errors. Can you describe more or paste complete error logs? Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
2023-04-26 18:47:26,311 - USING:: version = False Thanks for your quick reply, please see attached as the whole log from the service start. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @spencer03172023, Thanks for the logs. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your reply. correct, I use bundle Swin_unetr_btcv_sgmentation on the dataset as you mentioned. my steps as below:
two things I want to mentioned here, since in the first time I tried this, I got cuda out of memory error, then I set 'PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512' to fix this. another error I met is that "id" = "training" error, then I modified the code change train to training(I may post the detail later) after that, I can do active learning for training process, but seems no updated weights output and all I saw was the log I posted before |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the detailed post, @spencer03172023. This is strange. Can you please try the Segmentation model (https://github.com/Project-MONAI/MONAILabel/tree/main/sample-apps/radiology#segmentation), which also segments abdominal organs, and let us know if you experience the same issue? I'd like to check whether this is a bundle model issue. Please let us know. |
Beta Was this translation helpful? Give feedback.
-
Hi,
i follow the instructions from https://docs.monai.io/projects/label/en/latest/quickstart.html try to reproduce the active learning process.
But seems it was not run successfully with updated model.pt output and saved.
I post the log as below, anyone can help me?
[2023-04-26 18:49:44,974] [31098] [MainThread] [INFO] (monailabel.interfaces.app:492) - New label saved for: spleen_9 => spleen_9
[2023-04-26 18:49:45,026] [31098] [Thread-1] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2023-04-26 18:49:56,905] [31098] [MainThread] [INFO] (monailabel.utils.async_tasks.task:41) - Train request: {'model': 'swin_unetr_btcv_segmentation', 'device': 'cuda', 'pretrained': True, 'max_epochs': 50, 'val_split': 0.2, 'multi_gpu': True, 'gpus': 'all', 'tracking': 'None', 'tracking_uri': '', 'tracking_experiment_name': '', 'model_filename': 'model.pt', 'client_id': 'user-xyz', 'bundle_path': '/home/ubuntu/qiushenjie/monai/monaibundle/model/swin_unetr_btcv_segmentation'}
[2023-04-26 18:49:56,906] [31098] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:49) - Before:: /home/ubuntu/qiushenjie/monai:
[2023-04-26 18:49:56,906] [31098] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:53) - After:: /home/ubuntu/qiushenjie/monai::/home/ubuntu/qiushenjie/monai/monaibundle/model/swin_unetr_btcv_segmentation
[2023-04-26 18:49:56,907] [31098] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:65) - COMMAND:: /home/ubuntu/qiushenjie/monai/venv/bin/python -m monailabel.interfaces.utils.app -m train -r {"model":"swin_unetr_btcv_segmentation","device":"cuda","pretrained":true,"max_epochs":50,"val_split":0.2,"multi_gpu":true,"gpus":"all","tracking":"None","tracking_uri":"","tracking_experiment_name":"","model_filename":"model.pt","client_id":"user-xyz","bundle_path":"/home/ubuntu/qiushenjie/monai/monaibundle/model/swin_unetr_btcv_segmentation"}
[2023-04-26 18:49:57,161] [21773] [MainThread] [INFO] (main:37) - Initializing App from: /home/ubuntu/qiushenjie/monai/monaibundle; studies: /home/ubuntu/qiushenjie/monai/Task09_Spleen/imagesTr; conf: {'models': 'swin_unetr_btcv_segmentation'}
[2023-04-26 18:49:59,904] [21773] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2023-04-26 18:50:08,921] [21773] [MainThread] [INFO] (monailabel.utils.others.generic:303) - +++ Adding Bundle from Zoo: swin_unetr_btcv_segmentation => /home/ubuntu/qiushenjie/monai/monaibundle/model/swin_unetr_btcv_segmentation
[2023-04-26 18:50:08,921] [21773] [MainThread] [INFO] (monailabel.utils.others.generic:313) - +++ Using Bundle Models: ['swin_unetr_btcv_segmentation']
[2023-04-26 18:50:08,921] [21773] [MainThread] [INFO] (monailabel.interfaces.app:134) - Init Datastore for: /home/ubuntu/qiushenjie/monai/Task09_Spleen/imagesTr
[2023-04-26 18:50:08,921] [21773] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: False; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '.xml']
[2023-04-26 18:50:08,925] [21773] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2023-04-26 18:50:09,267] [21773] [MainThread] [INFO] (main:67) - +++ Adding Inferer:: swin_unetr_btcv_segmentation => <monailabel.tasks.infer.bundle.BundleInferTask object at 0x7f3b323a3490>
[2023-04-26 18:50:09,268] [21773] [MainThread] [INFO] (main:81) - +++ Adding Trainer:: swin_unetr_btcv_segmentation => <monailabel.tasks.train.bundle.BundleTrainTask object at 0x7f3b323a34d0>
[2023-04-26 18:50:09,268] [21773] [MainThread] [INFO] (main:91) - Active Learning Strategies:: ['random', 'first']
[2023-04-26 18:50:09,268] [21773] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /home/ubuntu/.cache/monailabel/sessions
[2023-04-26 18:50:09,268] [21773] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
usage: app.py [-h] [-a APP] [-s STUDIES] -m
{infer,train,info,batch_infer,scoring} [-r REQUEST] [-o OUTPUT]
[-d] [--multi_gpu] [--local_rank LOCAL_RANK]
optional arguments:
-h, --help show this help message and exit
-a APP, --app APP
-s STUDIES, --studies STUDIES
-m {infer,train,info,batch_infer,scoring}, --method {infer,train,info,batch_infer,scoring}
-r REQUEST, --request REQUEST
-o OUTPUT, --output OUTPUT
-d, --debug
--multi_gpu
--local_rank LOCAL_RANK
[2023-04-26 18:50:09,696] [31098] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:83) - Return code: 255
Beta Was this translation helpful? Give feedback.
All reactions