Loading an Atlas-Annotation is irritating

[SaibotMagd]

I try to load the 25um ABA atlas into mag:

bash = run + \
        " --img " + '/home/tgottsch/anaconda3/ClearMap2/ClearMap/Resources/Atlas' + \
        ' --reg_suffixes atlasVolume.nii annotation.nii' + \
        " --labels ABA_annotation.json" + \
        " --roi_profile atlas"
os.system(bash)

But as you can see it can't find the annotation file in the folder and so it can't bring the labels onto it.

So i run a "useless" registration:

bash = run + \
        " --img " + '/home/tgottsch/anaconda3/ClearMap2/ClearMap/Resources/Atlas/ABA_25um_atlasVolume.nii /home/tgottsch/anaconda3/ClearMap2/ClearMap/Resources/Atlas/ABA_25um_annotation.nii' + \
        " --register import_atlas " + \
        " --atlas_profile atlas,nosmooth"
os.system(bash)

After it finished I overwrite the new created atlasVolume.nii and annotation.nii in the newly created folder "ABA_25um_atlasVolume.nii_import" with the original once. When I load it now it works quite fine:

bash = run + \
        " --img " + '/home/tgottsch/anaconda3/ClearMap2/ClearMap/Resources/Atlas/ABA_25um_atlasVolume.nii_import' + \
        ' --reg_suffixes atlasVolume.nii annotation.nii' + \
        " --labels ABA_annotation.json" + \
        " --roi_profile atlas"
os.system(bash)

I know it's so wrong but it works. I don't know why this happen. I guess it could have something to do with the mag database file, but don't know.

[yoda-vid]

Thanks for pointing this out!

File loading does make some assumptions about filenames, which may not always work properly. What's strange is that the annotations do appear to have loaded in your first image, even though the file doesn't should up in the dropdown box.

I am trying to reproduce this on a folder that I think has a similar structure to yours, but it's working for me at least. The folder contains:

../../data/atlas_test:
annotation.nii  atlasVolume.nii labels.csv

which I am loading by:

./run.py ../../data/atlas_test --reg_suffixes atlasVolume.nii annotation.nii --labels ../../data/atlas_test/labels.csv

There could be something funny with how the files are being detected in your first scenario, and your second scenario probably works because the import puts the files in an expected structure. Could you post your console output? We can see how the files are being detected/loaded to troubleshoot.

[SaibotMagd]

Hm, I don't have a labels.csv file and I don't know how it should look like. I also don't know how to create json label file the ABA provided, but this will be a problem in the future. When I use the "region_ids.csv" instead it doesn't load at all:

root - INFO - set resolutions to [[20. 16. 16.]]
root - INFO - Loading image with SimpleITK: /home/tgottsch/Atlas_project/magellanmapper/examples/ADMBA-E18pt5/annotation.mhd
root - INFO - image scaling compared to image5d: [1. 1. 1.]
root - ERROR - Traceback (most recent call last):
root - ERROR -   File "/home/tgottsch/anaconda3/envs/mag/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2898, in get_loc
root - ERROR -     return self._engine.get_loc(casted_key)
root - ERROR -   File "pandas/_libs/index.pyx", line 70, in pandas._libs.index.IndexEngine.get_loc
root - ERROR -   File "pandas/_libs/index.pyx", line 101, in pandas._libs.index.IndexEngine.get_loc
root - ERROR -   File "pandas/_libs/hashtable_class_helper.pxi", line 1675, in pandas._libs.hashtable.PyObjectHashTable.get_item
root - ERROR -   File "pandas/_libs/hashtable_class_helper.pxi", line 1683, in pandas._libs.hashtable.PyObjectHashTable.get_item
root - ERROR - KeyError: 'id'
root - ERROR - 
The above exception was the direct cause of the following exception:
root - ERROR - Traceback (most recent call last):
root - ERROR -   File "/home/tgottsch/Atlas_project/magellanmapper/run.py", line 207, in <module>
root - ERROR -     main()
root - ERROR -   File "/home/tgottsch/Atlas_project/magellanmapper/run.py", line 139, in main
root - ERROR -     launch_magmap()
root - ERROR -   File "/home/tgottsch/Atlas_project/magellanmapper/run.py", line 124, in launch_magmap
root - ERROR -     cli.main()
root - ERROR -   File "/home/tgottsch/Atlas_project/magellanmapper/magmap/io/cli.py", line 853, in main
root - ERROR -     process_tasks()
root - ERROR -   File "/home/tgottsch/Atlas_project/magellanmapper/magmap/io/cli.py", line 745, in process_tasks
root - ERROR -     config.filename, series, offset, size, config.proc_type)
root - ERROR -   File "/home/tgottsch/Atlas_project/magellanmapper/magmap/io/np_io.py", line 374, in setup_images
root - ERROR -     labels_ref)
root - ERROR -   File "/home/tgottsch/Atlas_project/magellanmapper/magmap/atlas/ontology.py", line 185, in create_lookup_pd
root - ERROR -     ids = df[config.ABAKeys.ABA_ID.value]
root - ERROR -   File "/home/tgottsch/anaconda3/envs/mag/lib/python3.6/site-packages/pandas/core/frame.py", line 2906, in __getitem__
root - ERROR -     indexer = self.columns.get_loc(key)
root - ERROR -   File "/home/tgottsch/anaconda3/envs/mag/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2900, in get_loc
root - ERROR -     raise KeyError(key) from err
root - ERROR - KeyError: 'id'

[SaibotMagd]

I used the json-file instead and it worked. But I never saw a command without "--img" in the beginning. This is pretty new for me:

bash = run + \
        " ../examples/test" + \
        " --reg_suffixes atlasVolume.nii annotation.nii " + \
        " --labels ../examples/test/ontology17.json" + \
        " --roi_profile atlas"

Looks pretty good and the LUT is grey! So what is the difference between "--img" and non a parameter-tag?

[yoda-vid]

Hm, I don't have a labels.csv file and I don't know how it should look like. I also don't know how to create json label file the ABA provided, but this will be a problem in the future. When I use the "region_ids.csv" instead it doesn't load at all

Thanks for posting your stack trace. I can see now why loading the region_ids.csv is causing trouble. The region ID and name columns are named differently in that file, whereas the app expects the naming in the .json file. I'll fix the labels loader to be more tolerant of those headers.

In the meantime, you could change these headers in the region_ids.csv file so it's recognized:

• Region -> id
• RegionName -> name

So what is the difference between "--img" and non a parameter-tag?

Yeah this is new, where the --img parameter can be dropped if the image paths are placed at the start of the command. This simplifies the command slightly, and it also allows files to be loaded through the OS file browser (eg "Open with" in Finder or Windows Explorer; Linux is a bit more complicated and needs an application entry file to be added that I haven't worked on yet) when using the standalone app packages (the pre-releases right now).

I'm still trying to troubleshoot your initial issue where the registered images don't show up. Are there any console errors or warnings when you load the files that way?

[yoda-vid]

Also posted a PR (#27) with a fix for the labels reference file loading.

[yoda-vid]

@SaibotMagd are you seeing this loading issue anymore from your first comment? I haven't been able to reproduce it exactly, though I found a related issue for files with two extensions (eg .nii.gz) that I'm fixing.

[yoda-vid]

PR #64 fixes the .nii.gz, two-extensions case.