Support for .mib Files

[CSSFrancis]

Describe the functionality you would like to see.

Currently pyxem partially supports loading .mib files. There was an effort to increase the ability to load these type of files in pyxem/pyxem#732 but that never reached fruition. I think that it would be a good idea to port some of this loading here.

LiberTEM has a much more complete reader here and while I would like to not duplicate efforts too much it might be a good place to start/ it would be good to migrate some of the effort to a package with less dependencies.

Describe the context

This will support Merlin Detectors often used for 4-D STEM. These detectors are kind of unique in that often they are split detectors.

Additional information

Honestly, I have never used a Merlin detector/looked at data from a Merlin detector. I can transfer the loading capabilities from pyxem fairly easily. But I might need some additional information/help.

[jat255]

https://github.com/AndrewHerzing/merlintools and @AndrewHerzing immediately come to mind as potential sources of prior art in this area

[CSSFrancis]

It does seem like there are a lot of codes out there for loading .mib files. It does also seem like there are many different configurations which can be a challenge when writing a single writer for many formats. As long as there is interest/support I can take this up. It does seem like a large portion of pyxem's users would like expanded capabilities reguarding reading .mib files.

[jat255]

This would certainly help us move things along in NexusLIMS as well, as a .mib reader has been on our todo back-burner for quite a while. I could help test/review as needed

[ericpre]

Thanks @CSSFrancis, this should be indeed quite useful! I am putting the link to this comment, as I think this is good reference materials! :)

When I worked on the pyxem implementation when I had to process some of the merlin data, I find the structure not very convenient to work with and my impression is that it would be worth starting from a clean slate. The pyxem implementation have a few useful feature to deal with flyback, reshape of the data, add quadrant gap, etc. I suspect that some of it are useful for specific setup or early version of camera and this legacy makes the code unnecessarily complicated. This is why I would suggest to start from a clean structure and add more features later and understand/document when these feature are relevant i.e. what version of camera needs it - we need this for long-term maintenance.

[CSSFrancis]

Thanks @CSSFrancis, this should be indeed quite useful! I am putting the link to this comment, as I think this is good reference materials! :)

When I worked on the pyxem implementation when I had to process some of the merlin data, I find the structure not very convenient to work with and my impression is that it would be worth starting from a clean slate. The pyxem implementation have a few useful feature to deal with flyback, reshape of the data, add quadrant gap, etc. I suspect that some of it are useful for specific setup or early version of camera and this legacy makes the code unnecessarily complicated. This is why I would suggest to start from a clean structure and add more features later and understand/document when these feature are relevant i.e. what version of camera needs it - we need this for long-term maintenance.

I think a limited scope to start with is fair. As I don't have access to a camera I'm slightly limited but the number of different test datasets that I can acquire. If people do want their specific format supported then it probably depends on them adding it.

[ericpre]

Yes, sounds like good plan. There are small some files at https://github.com/matkraj/read_mib and also some code to create test files in pyxem/pyxem#732, which are possibly not necessary anymore?!

[magnunor]

I very much agree with having a .mib reader in rosettasciio, here in Trondheim we have both a single chip and quad chip MerlinEM so we could easily contribute different .mib files.

I also suspect Quantum Detectors (the vendor which sells MerlinEM) would be happy to assist with this, as they're generally very happy having open source code which can work with their data.

[jlaehne]

I also suspect Quantum Detectors (the vendor which sells MerlinEM) would be happy to assist with this, as they're generally very happy having open source code which can work with their data.

Yes, we should push vendors to contribute by at least using open formats like hdf5, for which readers are easily written, or to even support reader plugins themselves. For the CL community, we have two vendors now basically working on the contribution/upkeep of the plugins for their formats.

[sivborg]

As part of the Trondheim group, I am available for supplying small testing .mib files from single chip/quad chip MerlinEM. So let me know if I can be of assistance.

[ericpre]

Completed in #174. Follow up issue to support more features in #219.