Annotation job fails on _name_to_id - Digital Slide Archive (DSA) custom collection not available inside MONAILabel Server

[orenlivne]

I'm trying to annotate pathology images using monai within DSA. Any advice on fixing it would be greatly appreciated. Thank you.

Problem

1. An annotation (inference) MONAILabel server call from the DSA GUI fails inside _name_to_id, probably because the collection I created in DSA is not visible to the monailabel server (even though it is configured to look at the DSA API for sample data). When the infer method calls _name_to_id, it does not return anything inside the foreach loop, which returns an ID, name tuple. The fallback is returning the name only, causing the stack trace. This means _name_to_id has a small bug in its last line, but my real issue is the image I am passing from DSA to the monai server is not found in any folder. The problem does not occur for collections that come with DSA, such as TCGA, only for the new collection I created.
2. The DSA monai annotation is very slow (30-40s before it finally calls the infer function and crashes), even for a small ROI of 400x200 pixels. Could be related to the wrong image path issue.

Monailabel source dsa.py

    def _name_to_id(self, name):
        folders = self.folders if self.folders else self._get_all_folders()
        ###### I added the following two lines for debugging, cf. logs below:
        logger.info(f"name: {name}")
        logger.info(f"folders: {folders}")
        
        for folder in folders:
            data = self.gc.get("item", parameters={"folderId": folder, "limit": 0})
            ###### added this logging too
            logger.info(f"folder: {folder} data: {data}")
            for d in data:
                if d.get("largeImage") and d["name"] == name or Path(d["name"]).stem == name:
                    return d["_id"], d["name"].  ###### nothing must have been returned during this loop.**
        return name  ###### So this fallback is called and causes an exception since a tuple is expected by the caller.**

MONAILabel Server Stack Trace:

File "/Users/oren/monai/MONAILabel/monailabel/endpoints/wsi_infer.py", line 126, in api_run_wsi_inference
    return run_wsi_inference(background_tasks, model, image, session_id, None, wsi, output)
  File "/Users/oren/monai/MONAILabel/monailabel/endpoints/wsi_infer.py", line 110, in run_wsi_inference
    result = instance.infer_wsi(request)
  File "/Users/oren/monai/MONAILabel/monailabel/interfaces/app.py", line 633, in infer_wsi
    image = datastore.get_image_uri(request["image"])
  File "/Users/oren/monai/MONAILabel/monailabel/datastore/dsa.py", line 152, in get_image_uri
    image_id, name = self._name_to_id(image_id)
ValueError: too many values to unpack (expected 2)

Steps to Reproduce

Hardware: Apple Air, Apple M1 CPU, Monterey MacOS. Note that in the steps below two bugs specific to M1 + docker are fixed that I comment on, but are these are not relevant to the current discussion.

• Install OpenSlide: brew install openslide.
• Find the location of the openslide dylib:
  brew info openslide | grep Cellar | awk {'print $1}') and set to an environment variable. On Monterey, this is export OPENSLIDE_LIB=opt/homebrew/Cellar/openslide/3.4.1_7/lib
• Install DSA. Add the line chmod g+w /var/run/docker.sock 2>/dev/null to start_girder.sh (BUG 1). Run DSA via docker-compose.
• DSA is running on localhost:8080.
• Within a conda env, install monailabel from weekly release with pip install monailabel-weekly (note: the main release pip package is broken for Apple M1 due to bad package numpymaxflow==0.0.2 which leads numpy wheel building to fail; the weekly release upgraded to 0.0.5, which works).
• Download the monai pathology app.
• Note that DYLD_LIBRARY_PATH cannot be passed to a script if SIP is enabled on MacOS and thus OpenSlide won't be recognized by the monailabel server(BUG 2). Thus, start the monailabel server script with an additional line dynamically added to it that sets the library path, with the command tmpfile=$(mktemp /tmp/monailabel.XXXXXX) && ( cat $(which monailabel) | awk -v n=2 -v s="export DYLD_LIBRARY_PATH=\"${OPENSLIDE_LIB}\"" 'NR == n {print s} {print}' > ${tmpfile} ) && chmod +x ${tmpfile} && ( ${tmpfile} start_server --app $HOME/out/pathology --studies http://0.0.0.0:8080/api/v1 ) ; rm ${tmpfile}.
• Install the monai DSA plugin:
• Download the monai DSA plugin docker image: docker pull projectmonai/monailabel-dsa. Install it using the
  Slicer CLI Web under DSA admin plugins.
• Open the HistomicsUI, open image, select ROI, set
  ** MONAILabel Address: http://host.docker.internal:8000/ (note: if localhost:8080 is used here, the job will fail on Permission Denied; this is the appropriate docker address on Mac).
  ** Model Name: segmentation_nuclei.

DSA Job Log

Title:MONAILabel Annotations on sample-image.tiff
Type:projectmonai/monailabel-dsa:latest#MONAILabelAnnotation
Job ID:6391e73bf5e935c6e8bcb171
Status: ERROR
Timeline:
0 s163.411 s
Created:December 8, 2022 at 8:31:39
Scheduled start:December 8, 2022 at 8:31:39
Last update:December 8, 2022 at 8:34:22
Log output:
[2022-12-08 13:34:19,563] INFO: Running container: image: projectmonai/monailabel-dsa@sha256:99c7a31b0be9790205efa2327b38a8bf8583d4cab71a157ddbbdf0812e69f100 args: ['MONAILabelAnnotation', '--analysis_level', '0', '--analysis_roi', '14967, 19208, 451, 217', '--analysis_tile_size', '1024', '--api-url', 'http://girder:8080/api/v1/', '--extra_params', '{}', '--girder-token', 'd9C4qgeV3sEku4QT6xLM6ecMsQCABJ2gOLE2k2qmWLOWaOy5J99W1CSB6wicLwNG', '--min_fgnd_frac', '-1', '--min_poly_area', '80', '--model_name', 'segmentation_nuclei', '--server', 'http://host.docker.internal:8000/', '/mnt/girder_worker/2e6ee16dbe1f4d86abbc4b9cc8e32dba/sample-image.tiff', '/mnt/girder_worker/2e6ee16dbe1f4d86abbc4b9cc8e32dba/MONAILabel Annotations-outputAnnotationFile.anot'] runtime: None kwargs: {'tty': False, 'volumes': {'/var/folders/zw/c2j1_twn3_g09ldg5frh0pmm0000gn/T/tmpjb3ekwrq': {'bind': '/mnt/girder_worker/2e6ee16dbe1f4d86abbc4b9cc8e32dba', 'mode': 'rw'}}, 'detach': True, 'network': 'container:4ec6011efa66054d0e9310701f107d5019de73d95f08a651484582a0f640dd67'}
INFO:root:CLI Parameters ...

INFO:root:USING:: inputImageFile = /mnt/girder_worker/2e6ee16dbe1f4d86abbc4b9cc8e32dba/sample-image.tiff
INFO:root:USING:: outputAnnotationFile = /mnt/girder_worker/2e6ee16dbe1f4d86abbc4b9cc8e32dba/MONAILabel Annotations-outputAnnotationFile.anot
INFO:root:USING:: analysis_level = 0
INFO:root:USING:: analysis_roi = [14967.0, 19208.0, 451.0, 217.0]
INFO:root:USING:: analysis_tile_size = 1024.0
INFO:root:USING:: girderApiUrl = http://girder:8080/api/v1/
INFO:root:USING:: extra_params = {}
INFO:root:USING:: girderToken = d9C4qgeV3sEku4QT6xLM6ecMsQCABJ2gOLE2k2qmWLOWaOy5J99W1CSB6wicLwNG
INFO:root:USING:: min_fgnd_frac = -1.0
INFO:root:USING:: min_poly_area = 80.0
INFO:root:USING:: model_name = segmentation_nuclei
INFO:root:USING:: server = http://host.docker.internal:8000/
INFO:root:>> Reading input image ... 

INFO:root:Run MONAILabel Task... and collect the annotations: [14967.0, 19208.0] => [451.0, 217.0]
INFO:root:For Server Logs click/open:  http://host.docker.internal:8000/logs/?refresh=3
Traceback (most recent call last):
  File "/opt/monailabel/dsa/cli/MONAILabelAnnotation/MONAILabelAnnotation.py", line 174, in <module>
    main(CLIArgumentParser().parse_args())
  File "/opt/monailabel/dsa/cli/MONAILabelAnnotation/MONAILabelAnnotation.py", line 168, in main
    fetch_annotations(args, tiles)
  File "/opt/monailabel/dsa/cli/MONAILabelAnnotation/MONAILabelAnnotation.py", line 55, in fetch_annotations
    _, res = client.wsi_infer(model=args.model_name, image_in=image, body=body, output=output)
  File "/opt/monailabel/dsa/cli/client.py", line 188, in wsi_infer
    raise MONAILabelClientException(
cli.client.MONAILabelClientException: (2, "Status: 500; Response: b'Internal Server Error'")
DockerException: Non-zero exit code from docker container (1).
  File "/opt/venv/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/slicer_cli_web/slicer_cli_web/girder_worker_plugin/direct_docker_run.py", line 87, in __call__
    super().__call__(*args, **kwargs)
  File "/opt/girder_worker/girder_worker/docker/tasks/__init__.py", line 337, in __call__
    super(DockerTask, self).__call__(*args, **kwargs)
  File "/opt/girder_worker/girder_worker/task.py", line 154, in __call__
    results = super(Task, self).__call__(*_t_args, **_t_kwargs)
  File "/opt/venv/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/slicer_cli_web/slicer_cli_web/girder_worker_plugin/direct_docker_run.py", line 123, in run
    return _docker_run(task, **kwargs)
  File "/opt/girder_worker/girder_worker/docker/tasks/__init__.py", line 405, in _docker_run
    _run_select_loop(task, container, read_streams, write_streams)
  File "/opt/girder_worker/girder_worker/docker/tasks/__init__.py", line 251, in _run_select_loop
    raise DockerException('Non-zero exit code from docker container (%d).' % exit_code)
Keyword arguments:
{
  "container_args": [
    "MONAILabelAnnotation",
    "--analysis_level",
    "0",
    "--analysis_roi",
    "14967, 19208, 451, 217",
    "--analysis_tile_size",
    "1024",
    "--api-url",
    "<slicer_cli_web.girder_worker_plugin.direct_docker_run.GirderApiUrl object at 0x41958da550>",
    "--extra_params",
    "{}",
    "--girder-token",
    "<slicer_cli_web.girder_worker_plugin.direct_docker_run.GirderToken object at 0x42891d5280>",
    "--min_fgnd_frac",
    "-1",
    "--min_poly_area",
    "80",
    "--model_name",
    "segmentation_nuclei",
    "--server",
    "http://host.docker.internal:8000/",
    "<slicer_cli_web.girder_worker_plugin.direct_docker_run.DirectGirderFileIdToVolume: File ID=6391ba8af5e935c6e8bcb152 -> \"sample-image.tiff\">",
    "<girder_worker.docker.transforms.VolumePath: \"MONAILabel Annotations-outputAnnotationFile.anot\">"
  ],
  "image": "projectmonai/monailabel-dsa@sha256:99c7a31b0be9790205efa2327b38a8bf8583d4cab71a157ddbbdf0812e69f100",
  "pull_image": "if-not-present"
}

MONAILabel Server Logs

[2022-12-08 08:31:09,012] [60736] [MainThread] [INFO] (monailabel.interfaces.app:468) - App Init - completed
[2022-12-08 08:31:09,012] [60736] [MainThread] [INFO] (timeloop:60) - Starting Timeloop..
[2022-12-08 08:31:09,013] [60736] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete.<locals>.run_scheduler at 0x16b3d7e20>
[2022-12-08 08:31:09,013] [60736] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set
[2022-12-08 08:34:22,032] [60736] [MainThread] [INFO] (monailabel.endpoints.wsi_infer:108) - WSI Infer Request: {'model': 'segmentation_nuclei', 'image': 'sample-image', 'output': 'dsa', 'level': 0, 'location': [14967, 19208], 'size': [451, 217], 'tile_size': [1024, 1024], 'min_poly_area': 80, 'wsi_tiles': []}
[2022-12-08 08:34:22,165] [60736] [MainThread] [INFO] (monailabel.datastore.dsa:138) - name: sample-image
[2022-12-08 08:34:22,165] [60736] [MainThread] [INFO] (monailabel.datastore.dsa:139) - folders: ['63865dff31e1eb24754d2e99', '63865dd1f37a980298fa9ce9']
[2022-12-08 09:42:21,506] [64332] [MainThread] [INFO] (monailabel.datastore.dsa:143) - folder: 63865dff31e1eb24754d2e99 data: [{'_id': '63866186
31e1eb24754d2eb0', '_modelType': 'item', 'baseParentId': '63865dff31e1eb24754d2e98', 'baseParentType': 'collection', 'created': '2022-11-29T19:4
6:14.565000+00:00', 'creatorId': '63865dd0f37a980298fa9ce4', 'description': '', 'folderId': '63865dff31e1eb24754d2e99', 'meta': {}, 'name': 'Eas
y1.png', 'size': 2899365, 'updated': '2022-11-29T19:46:14.565000+00:00'}, {'_id': '6386618a31e1eb24754d2eb3', '_modelType': 'item', 'baseParentI
d': '63865dff31e1eb24754d2e98', 'baseParentType': 'collection', 'created': '2022-11-29T19:46:18.376000+00:00', 'creatorId': '63865dd0f37a980298f
a9ce4', 'description': '', 'folderId': '63865dff31e1eb24754d2e99', 'largeImage': {'fileId': '6386619c31e1eb24754d2eb5', 'sourceName': 'tifffile'
}, 'meta': {}, 'name': 'G10-3_pelvis_crop.tif', 'size': 35417073, 'updated': '2022-11-29T19:46:18.376000+00:00'}, {'_id': '6386619e31e1eb24754d2
eb6', '_modelType': 'item', 'baseParentId': '63865dff31e1eb24754d2e98', 'baseParentType': 'collection', 'created': '2022-11-29T19:46:38.044000+0
0:00', 'creatorId': '63865dd0f37a980298fa9ce4', 'description': '', 'folderId': '63865dff31e1eb24754d2e99', 'largeImage': {'fileId': '638661ae31e
1eb24754d2eb8', 'sourceName': 'tiff'}, 'meta': {}, 'name': 'Huron.Image2_JPEG2K.tif', 'size': 31529726, 'updated': '2022-11-29T19:46:38.044000+0
0:00'}, {'_id': '638661b031e1eb24754d2eb9', '_modelType': 'item', 'baseParentId': '63865dff31e1eb24754d2e98', 'baseParentType': 'collection', 'c
reated': '2022-11-29T19:46:56.641000+00:00', 'creatorId': '63865dd0f37a980298fa9ce4', 'description': '', 'folderId': '63865dff31e1eb24754d2e99',
 'largeImage': {'fileId': '638661d331e1eb24754d2ebf', 'sourceName': 'openslide'}, 'meta': {}, 'name': 'JP2K-33003-1.svs', 'size': 63847265, 'upd
ated': '2022-11-29T19:46:56.641000+00:00'}, {'_id': '638661d731e1eb24754d2ec2', '_modelType': 'item', 'baseParentId': '63865dff31e1eb24754d2e98'
, 'baseParentType': 'collection', 'created': '2022-11-29T19:47:35.363000+00:00', 'creatorId': '63865dd0f37a980298fa9ce4', 'description': '', 'fo
lderId': '63865dff31e1eb24754d2e99', 'meta': {}, 'name': 'TCGA-02-0010-01Z-00-DX4.07de2e55-a8fe-40ee-9e98-bcb78050b9f7.svs', 'size': 0, 'updated
': '2022-11-29T19:47:35.363000+00:00'}]                                 
[2022-12-08 09:42:21,542] [64332] [MainThread] [INFO] (monailabel.datastore.dsa:143) - folder: 63865dd1f37a980298fa9ce9 data: []

Note the logging I added, showing the name and folders inside _name_to_id. Note that the new collection I added (the second in the list above; the first is the TCGA data set that comes with DSA) has an empty data.

MONAILabel Server Screen Output

[2022-12-08 08:31:09,013] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set                           
[2022-12-08 08:31:09,013] [60736] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set 
[2022-12-08 08:31:09,013] [60736] [MainThread] [INFO] (uvicorn.error:59) - Application startup complete.                            
[2022-12-08 08:31:09,013] [60736] [MainThread] [INFO] (uvicorn.error:206) - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to 
quit)                                                                                                                               
[2022-12-08 08:34:22,032] [60736] [MainThread] [INFO] (monailabel.endpoints.wsi_infer:108) - WSI Infer Request: {'model': 'segmentat
ion_nuclei', 'image': 'sample-image', 'output': 'dsa', 'level': 0, 'location': [14967, 19208], 'size': [451, 217], 'tile_size': [102
4, 1024], 'min_poly_area': 80, 'wsi_tiles': []}
[2022-12-08 08:34:22,165] [60736] [MainThread] [INFO] (monailabel.datastore.dsa:138) - name: sample-image                  
[2022-12-08 08:34:22,165] [60736] [MainThread] [INFO] (monailabel.datastore.dsa:139) - folders: ['63865dff31e1eb24754d2e99', '63865d
d1f37a980298fa9ce9']                                                                                                                
[2022-12-08 08:34:22,227] [60736] [MainThread] [ERROR] (uvicorn.error:369) - Exception in ASGI application
Traceback (most recent call last):                                                                                                  
  File "/Users/oren/monai/MONAILabel/monailabel/datastore/dsa.py", line 150, in get_image_uri
    name = self.get_image_info(image_id)["name"]                                                                                    
  File "/Users/oren/monai/MONAILabel/monailabel/datastore/dsa.py", line 176, in get_image_info
    return self.gc.getItem(image_id)  # type: ignore                                                                                
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/girder_client/__init__.py", line 619, in getItem
    return self.getResource('item', itemId)                                                                                         
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/girder_client/__init__.py", line 519, in getResource
    return self.get(route)                                                                                                          
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/girder_client/__init__.py", line 471, in get
    return self.sendRestRequest('GET', path, parameters, jsonResp=jsonResp)                                   
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/girder_client/__init__.py", line 463, in sendRestRequest
    raise HttpError(                                                                                                                
girder_client.HttpError: HTTP error 400: GET http://0.0.0.0:8080/api/v1/item/sample-image
Response text: {"field": "id", "message": "Invalid ObjectId: sample-image", "type": "validation"}                              
 During handling of the above exception, another exception occurred:                                        
                                                                                                                                    
Traceback (most recent call last):                                                                                                  
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi
    result = await app(self.scope, self.receive, self.send)                                                                         
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/fastapi/applications.py", line 269, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/starlette/exceptions.py", line 93, in __call__
    raise exc
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/starlette/exceptions.py", line 82, in __call__
    await self.app(scope, receive, sender)
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/starlette/routing.py", line 670, in __call__
    await route.handle(scope, receive, send)
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/starlette/routing.py", line 266, in handle
    await self.app(scope, receive, send)
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/starlette/routing.py", line 65, in app
    response = await func(request)
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/fastapi/routing.py", line 227, in app
    raw_response = await run_endpoint_function(
  File "/Users/oren/miniconda3/envs/monai2/lib/python3.10/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
    return await dependant.call(**values)
  File "/Users/oren/monai/MONAILabel/monailabel/endpoints/wsi_infer.py", line 126, in api_run_wsi_inference
    return run_wsi_inference(background_tasks, model, image, session_id, None, wsi, output)
  File "/Users/oren/monai/MONAILabel/monailabel/endpoints/wsi_infer.py", line 110, in run_wsi_inference
    result = instance.infer_wsi(request)
  File "/Users/oren/monai/MONAILabel/monailabel/interfaces/app.py", line 633, in infer_wsi
    image = datastore.get_image_uri(request["image"])
  File "/Users/oren/monai/MONAILabel/monailabel/datastore/dsa.py", line 152, in get_image_uri
    image_id, name = self._name_to_id(image_id)
ValueError: too many values to unpack (expected 2)