Every time the model creates new tranforms in cache folder, is there a way to stop it, because it take too much time

[aaekay]

Hi, I am a medical doctor and I am training a dataset on the deepgrow_3d model from scratch and with my annotations. I want to train the data in one go, instead of training it one by one, but every time I run the mailable server, it re-run the transformation on each patient CT, and it is taking too much of time. Is there a way to avoid it? Please tell us ways to train the dataset efficiently

[SachidanandAlle]

Using cached transform normally help to precompute any pre processing and reuse for every epoch.. but if you are using first transform after loading the image which depends on randomness then cache won't help much..

May be you can share some snippet to understand what is a happening at your end

Also you can train a model directly.. I mean just run as main function in your app.. it need not to be via rest api.. that can help you debug more..

[aaekay]

i am now running the main function directly. It is taking time just for transform to happen. Is there a way where I can save the transformations and call the data loader from the transformations only.
[2023-06-08 17:09:11] [1863504] [MainThread] [INFO] (root:108) - Image (transform): torch.Size([331, 331, 324]); Label: torch.Size([331, 331, 324])
[2023-06-08 17:09:11] [1863504] [MainThread] [INFO] (root:113) - Image (final): (324, 331, 331); Label: (324, 331, 331)
[2023-06-08 17:09:23] [1863504] [MainThread] [INFO] (root:261) - 17 => Image Shape: (324, 331, 331) => 1; Label Shape: (324, 331, 331) => 1; Unique Labels: 1
[2023-06-08 17:09:23] [1863504] [MainThread] [INFO] (root:103) - Image: /home/amit_g/scratch_old/datasets/tsdata/s0664.nii.gz; Label: /home/amit_g/scratch_old/datasets/tsdata/labels/final/s0664.nii.gz
[2023-06-08 17:09:48] [1863504] [MainThread] [INFO] (root:108) - Image (transform): torch.Size([498, 498, 607]); Label: torch.Size([498, 498, 607])
[2023-06-08 17:09:48] [1863504] [MainThread] [INFO] (root:113) - Image (final): (607, 498, 498); Label: (607, 498, 498)
[2023-06-08 17:11:25] [1863504] [MainThread] [INFO] (root:261) - 18 => Image Shape: (607, 498, 498) => 1; Label Shape: (607, 498, 498) => 1; Unique Labels: 1
[2023-06-08 17:11:25] [1863504] [MainThread] [INFO] (root:103) - Image: /home/amit_g/scratch_old/datasets/tsdata/s0210.nii.gz; Label: /home/amit_g/scratch_old/datasets/tsdata/labels/final/s0210.nii.gz