Memory-mapping is needed (or preferred); even if raw data can load to RAM?

[masa6161]

Hi. I've used the Matlab ver. of CaIman. However, I finally migrated from the Matlab version to the Python version.
I'm confused by some of the minor differences, but my main question is about memory mapping.

Is this necessary (or even preferable) if all the data to be processed can be loaded into RAM?
I've been experimenting with the source extraction by rewriting the demo_caiman_basic.py, but so far it is taking up a lot of time, storage space (17.6GB RAW tiffs vs 70.3 GB memmapped files), and the network overhead (cause my data are on network storage servers) in the memory-mapping.

As my dataset occasionally has a small number of neurons, I think the source extraction with full FOV is preferred. And these are already motion-corrected.
If I can perform the extraction without the memory-mapping, how can I do it? Otherwise, is there a better way?
I wonder if the none memmap will be achieved when I switched the cnmf_stride and rf parameters to None, but the program worked to make memory-mapped files.

Thanks to the development of a great tool and its community.
Best.

[EricThomson]

Good questions. As currently designed, the memory mapping is a required step.

The blow-up in size happens partially because of conversion to float: Caiman does lots of floating point operations on the images. We have had some discussion of whether we could get away with a lower bit-depth representation but haven't pushed hard on this (in particular see #1037).

If your data become too large to handle, I recommend switching to the online algorithms. For a related older discussion also see #750

[kushalkolar]

Run it locally instead of on the remote filesystem and then copy over to the remote filesystem when done?

[masa6161]

Run it locally instead of on the remote filesystem and then copy over to the remote filesystem when done?
Yeah, that's a good alternative way, I think so and try it too!
The network and read/write overhead in our 10Gbps LAN and RAID0 storage might be a little part of this when I am looking for the processing with a task manager, sorry.
The volume of the memmaped files is problematic cause the space of storage is limited.

[masa6161]

Additional question:
My dataset 5 separated tiffs; however, "6" mapped files exist.
About these 6 memmaped files,
1, set of 5 memmap files and; [basename]0000_[image dims][order]frames[total num frames/5],...,[basename]0004[image dims][order]frames[total num frames/5]
and
2: [basename][image dims]_[order]frames[total num of frames]
Are 1 and 2 similar or not? Sizes of 1 and 2 are almost equal.
How these file set(s) are utilized?

[EricThomson]

@masa6161 could you please open a new discussion about this question? I don't actually know the answer off the top of my head, but it is a different enough question it would be useful to have as a separate topic to pop up in searches.

[masa6161]

@EricThomson Sure. I'll post this as another topic of discussion. Thank you!