Add support for pixel trigger acquisition in quantumdetector reader

[ericpre]

Fix first item of #219.

Progress of the PR

• Use timestamps to get navigation shape when the navigation shape is not available otherwise (acquisition with pixel trigger and scanshape not in metadata),
• update docstring,
• update user guide,
• add a changelog entry in the upcoming_changes folder (see upcoming_changes/README.rst),
• Check formatting of the changelog entry (and eventual user guide changes) in the docs/readthedocs.org:rosettasciio build of this PR (link in github checks)
• add tests,
• ready for review.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.39%. Comparing base (0646b2c) to head (468482c).
Report is 16 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #235      +/-   ##
==========================================
+ Coverage   86.36%   86.39%   +0.03%     
==========================================
  Files          83       83              
  Lines       10604    10632      +28     
  Branches     2307     2316       +9     
==========================================
+ Hits         9158     9186      +28     
  Misses        930      930              
  Partials      516      516              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ericpre]

@magnunor, since you mentioned that you are "very interested" in this feature (#219 (comment)), can you please review it? If not, I will merge it.

[magnunor]

@ericpre, very nice! I'll have a look at this, and test it on some of my datasets. Probably tomorrow.

[magnunor]

I tested this on a number of datasets, and it seems to work nicely! In the current main branch, it gives the shape (65536, 256, 256), while with this pull request it gives (256, 256, 256, 256).

[magnunor]

I have some more feedback, but I won't have time to do it before Thursday.

[magnunor]

One thing which is missing is either i) a better error message when the navigation shape is missing, and the automatic navigation-shape-guesser doesn't work, or ii) better handling of mib-files where the navigation shape is unknown.

For example, I have file where the HDR-file says 'Frames in Acquisition (Number)': '0', and where there are no flyback pixels. Currently the error message is a bit cryptic: ValueError: cannot reshape array of size 0 into shape (256,256)

For ii), it should be possible to estimate the number of frames based on the size of the file, header size, number of chips and dtype. It could then return a 3D array.

[ericpre]

Thanks @magnunor for the review. If this PR doesn't introduce any regression, I think that it would be useful to merge it to include it in the next release.
I replied to your comments below, some of these may need more time to understand and document the use case and get small test files. I will try to deal with the 'Frames in Acquisition (Number)': '0' case, as this should be fairly straightforward even if this is not clear why it would occur.

One thing which is missing is either i) a better error message when the navigation shape is missing, and the automatic navigation-shape-guesser doesn't work, or ii) better handling of mib-files where the navigation shape is unknown.

Most of the time, the navigation_shape is not known. It should be included explicitly in the hdr file generated by recent version of the software (assuming that this feature has been released).
Are you aware of there other cases where estimating the navigation shape doesn't work (other than the one that you already mentioned: the 'Frames in Acquisition (Number)': '0' case)?

For example, I have file where the HDR-file says 'Frames in Acquisition (Number)': '0', and where there are no flyback pixels. Currently the error message is a bit cryptic: ValueError: cannot reshape array of size 0 into shape (256,256)

Do you know in what situation does this occur? I am wondering if there is some buggy/corrupted file or if this actually corresponding to a specific use case. If so, it would be useful to add a comment about it in the code.

Assuming this should be with single frame files, there are already some of these file in the test suite and the tests are still passing?

For ii), it should be possible to estimate the number of frames based on the size of the file, header size, number of chips and dtype. It could then return a 3D array.

The number of frames is already known from the code, how do you think this information should be used for reshaping purposes?

[ericpre]

@magnunor, the 'Frames in Acquisition (Number)': '0' case should now be covered, do you want to have another look?

[magnunor]

Thanks @magnunor for the review. If this PR doesn't introduce any regression, I think that it would be useful to merge it to include it in the next release.

Based on testing this PR on a number of datasets, I don't think there are any regressions.

Are you aware of there other cases where estimating the navigation shape doesn't work (other than the one that you already mentioned: the 'Frames in Acquisition (Number)': '0' case)?

One case could be where flyback was not used, meaning all the pixels have the same frame time.

Another case could be where the acquisition was interrupted. On some older datasets, it seems like the HDR file has a higher number ('Frames in Acquisition (Number)':) than the actual data in the MIB-file. Here is an example of such a file: https://filesender.sikt.no/?s=download&token=91e8ff4b-dda8-4cb8-997b-0504d7edaca2 (the link will expire 20. april)

Do you know in what situation does this occur? I am wondering if there is some buggy/corrupted file or if this actually corresponding to a specific use case. If so, it would be useful to add a comment about it in the code.

Assuming this should be with single frame files, there are already some of these file in the test suite and the tests are still passing?

I am actually not sure when the 'Frames in Acquisition (Number)': '0' occurs. The file is not single-frame. This is a dataset from 2018, so an older version of the Merlin software.

The number of frames is already known from the code, how do you think this information should be used for reshaping purposes?

Looking a bit closer at this, it seems like the part which is checking for the number of frames in the load_mib_data is not catching some of the cases where the acquisition is interrupted. Essentially, this part of the code is not being used for the file linked earlier:

rosettasciio/rsciio/quantumdetector/_api.py

Line 276 in 4f2d78e

elif number_of_frames_to_load < frame_number:

---

However, these issues also happen in the current main branch. So this is not a regression.

[ericpre]

Are you aware of there other cases where estimating the navigation shape doesn't work (other than the one that you already mentioned: the 'Frames in Acquisition (Number)': '0' case)?

One case could be where flyback was not used, meaning all the pixels have the same frame time.

Okay, but in that case, this is no way to estimate the navigation shape and it needs to be provided by the users. By default the current implementation will return a navigation_shape of (number_of_frames, ).

Another case could be where the acquisition was interrupted. On some older datasets, it seems like the HDR file has a higher number ('Frames in Acquisition (Number)':) than the actual data in the MIB-file. Here is an example of such a file: https://filesender.sikt.no/?s=download&token=91e8ff4b-dda8-4cb8-997b-0504d7edaca2 (the link will expire 20. april)

Do you know in what situation does this occur? I am wondering if there is some buggy/corrupted file or if this actually corresponding to a specific use case. If so, it would be useful to add a comment about it in the code.
Assuming this should be with single frame files, there are already some of these file in the test suite and the tests are still passing?

I am actually not sure when the 'Frames in Acquisition (Number)': '0' occurs. The file is not single-frame. This is a dataset from 2018, so an older version of the Merlin software.

I checked the merlin manual, it seems that it may correspond to a case where the acquisition is set to be "continuous and indefinite". The wording is not exactly as the one of used in the hdr file but it seems to make sense to a certain extent!

The number of frames is already known from the code, how do you think this information should be used for reshaping purposes?

Looking a bit closer at this, it seems like the part which is checking for the number of frames in the load_mib_data is not catching some of the cases where the acquisition is interrupted. Essentially, this part of the code is not being used for the file linked earlier:

rosettasciio/rsciio/quantumdetector/_api.py

Line 276 in 4f2d78e

elif number_of_frames_to_load < frame_number:

However, these issues also happen in the current main branch. So this is not a regression.

This was a bug in the code to reshape in case of interrupted acquisition - this should be fixed now. Can you confirm that the dataset you shared here have 5 pixels per line?

[magnunor]

This should be ready for merging now! Very nice improvements to the MIB-reader :)

---

Okay, but in that case, this is no way to estimate the navigation shape and it needs to be provided by the users. By default the current implementation will return a navigation_shape of (number_of_frames, ).

Indeed! Ignore the things I was writing about that, due to the bug you fixed this is no longer relevant.

This was a bug in the code to reshape in case of interrupted acquisition - this should be fixed now. Can you confirm that the dataset you shared here have 5 pixels per line?

I think for this specific dataset, things are a bit strange. These frames should be from a single line. But the times_diff - np.mean(times_diff) gives:

array([-8.66666667, -8.66666667, -8.66666667, -8.66666667,  1.33333333,
        1.33333333,  1.33333333,  1.33333333,  1.33333333,  1.33333333,
        1.33333333,  1.33333333,  1.33333333,  1.33333333,  1.33333333,
        1.33333333,  1.33333333,  1.33333333,  1.33333333,  1.33333333,
        1.33333333,  1.33333333,  1.33333333,  1.33333333,  1.33333333,
        1.33333333,  1.33333333,  1.33333333,  1.33333333,  1.33333333])

But I don't expect the file reader to be able to handle the navigation shape in strange datasets such as these. What was annoying earlier was that the data wasn't loading at all, but with the bug fix it loads just nicely :)

[ericpre]

Okay, thank you, I fixed the typos in the test name, we should be all good now!