Patch zarr reader for ome-zarr v0.4 plates

[DavidStirling]

Format for the indexing keys differs from older versions e.g. column_index -> columnIndex

[sbesson]

The changes are minimal and match my understanding of the evolution of the NGFF specification as well as bioformats2raw. Is there a way to test this functionally against a plate generated by a recent version of bioformats2raw?

[sbesson]

For completeness, I would slightly update this coment.
https://ngff.openmicroscopy.org/0.2/index.html#plate-md does not mandate any metadata for the row/column index. The keys detected by this utility were populated by bioformats2raw 0.4.0

In version 0.4 of the NGFF specification, columnIndex/rowIndex are required attributes for each element of the wells array

[DavidStirling]

@sbesson How about this?

Functional testing is difficult until we compile this into a full CellProfiler build. At that point we should be able to test by just dropping a .zarr root folder into the CP GUI.

[emilroz]

@DavidStirling we have a wheel built as a part of the process so we should be able to test the CLI no?

[DavidStirling]

You're right, but the CLI isn't much use without the main program modules. Perhaps install the wheel with the latest gs cellprofiler version and run it from there?

[emilroz]

Yes, please.

[DavidStirling]

Okay, running locally you need to drop zarrs and disable the "Images only" filter in the "Images" module, then add a random module to the pipeline. You should get this error when trying to read data from the v0.4 zarr:

On my end the PR fixes it and data reads as normal.

Running metadata extraction also works normally too. CP finds 464 planes in the v0.2 zarr and 261 in the v0.4 zarr, which would be in keeping with the sparse well population.

[emilroz]

Test command: cellprofiler -r -c --run-headless --log-level=DEBUG -p pipe.cppipe -o . -f 1 -l 1 --data-file=image_list.csv

Test with zarr.v2:

Reading c=2, z=0, t=0, series=0, index=None, XYWH=None
Getting image reader for: None, None, file:/root/97998_v0.2.zarr
Getting image reader for: AF647, None, file:/root/97998_v0.2.zarr
Reading c=2, z=0, t=0, series=0, index=None, XYWH=None
Getting image reader for: AF594, None, file:/root/97998_v0.2.zarr
Reading c=1, z=0, t=0, series=0, index=None, XYWH=None
Getting image reader for: None, None, file:/root/97998_v0.2.zarr
Getting image reader for: DAPI, None, file:/root/97998_v0.2.zarr
Reading c=0, z=0, t=0, series=0, index=None, XYWH=None
Getting image reader for: None, None, file:/root/97998_v0.2.zarr
Mon Dec  2 14:16:14 2024: Image # 1, module LoadData # 1: CPU_time = 0.19 secs, Wall_time = 0.24 secs
Mon Dec  2 14:16:14 2024: Image # 1, module RescaleIntensity # 2: CPU_time = 0.01 secs, Wall_time = 0.01 
Loading model Versatile (fluorescent nuclei)
Found model '2D_versatile_fluo' with alias 'Versatile (fluorescent nuclei)' for 'StarDist2D'.
Searching for  /root/.keras/models/StarDist2D/2D_versatile_fluo
Using cached model Versatile (fluorescent nuclei)
Loading network weights from 'weights_best.h5'.
Loading thresholds from 'thresholds.json'.
Using default values: prob_thresh=0.479071, nms_thresh=0.3.
Mon Dec  2 14:16:14 2024: Image # 1, module RunStarDist # 3: CPU_time = 12.72 secs, Wall_time = 6.40 secs
Mon Dec  2 14:16:20 2024: Image # 1, module FilterObjects # 4: CPU_time = 0.22 secs, Wall_time = 0.21 secs
Mon Dec  2 14:16:21 2024: Image # 1, module MeasureObjectSizeShape # 5: CPU_time = 0.93 secs, Wall_time = 0.94 secs
Mon Dec  2 14:16:22 2024: Image # 1, module FilterObjects # 6: CPU_time = 0.12 secs, Wall_time = 0.12 secs
Mon Dec  2 14:16:22 2024: Image # 1, module MeasureObjectIntensity # 7: CPU_time = 0.26 secs, Wall_time = 0.26 secs
Mon Dec  2 14:16:22 2024: Image # 1, module OverlayOutlines # 8: CPU_time = 0.04 secs, Wall_time = 0.04 secs
Mon Dec  2 14:16:22 2024: Image # 1, module SaveImages # 9: CPU_time = 0.01 secs, Wall_time = 0.01 secs
Mon Dec  2 14:16:22 2024: Image # 1, module ExportToSpreadsheet # 10: CPU_time = 0.00 secs, Wall_time = 0.00 secs
Closing reader <cellprofiler_core.utilities.zarr.ZarrReader object at 0x7f8c060b0cd0>

Test with zarr.v4:

Getting image reader for: AF647, None, file:/root/97998_v0.4.zarr
Reading c=2, z=0, t=0, series=0, index=None, XYWH=None
Getting image reader for: None, None, file:/root/97998_v0.4.zarr
Getting image reader for: AF647, None, file:/root/97998_v0.4.zarr
Reading c=2, z=0, t=0, series=0, index=None, XYWH=None
Getting image reader for: AF594, None, file:/root/97998_v0.4.zarr
Reading c=1, z=0, t=0, series=0, index=None, XYWH=None
Getting image reader for: None, None, file:/root/97998_v0.4.zarr
Getting image reader for: DAPI, None, file:/root/97998_v0.4.zarr
Reading c=0, z=0, t=0, series=0, index=None, XYWH=None
Getting image reader for: None, None, file:/root/97998_v0.4.zarr
Mon Dec  2 14:21:19 2024: Image # 1, module RunStarDist # 3: CPU_time = 12.48 secs, Wall_time = 6.24 secs
Mon Dec  2 14:21:25 2024: Image # 1, module FilterObjects # 4: CPU_time = 0.19 secs, Wall_time = 0.21 secs
Mon Dec  2 14:21:25 2024: Image # 1, module MeasureObjectSizeShape # 5: CPU_time = 0.94 secs, Wall_time = 0.94 secs
Mon Dec  2 14:21:26 2024: Image # 1, module FilterObjects # 6: CPU_time = 0.11 secs, Wall_time = 0.12 secs
Mon Dec  2 14:21:26 2024: Image # 1, module MeasureObjectIntensity # 7: CPU_time = 0.26 secs, Wall_time = 0.26 secs
Mon Dec  2 14:21:26 2024: Image # 1, module OverlayOutlines # 8: CPU_time = 0.04 secs, Wall_time = 0.04 secs
Mon Dec  2 14:21:26 2024: Image # 1, module SaveImages # 9: CPU_time = 0.01 secs, Wall_time = 0.01 secs
Mon Dec  2 14:21:26 2024: Image # 1, module ExportToSpreadsheet # 10: CPU_time = 0.00 secs, Wall_time = 0.00 secs
Closing reader <cellprofiler_core.utilities.zarr.ZarrReader object at 0x7f76b9071ca0>