Which is the essential memory mapped file for the situation of signal extraction?

[masa6161]

Dear CaImAn developers and users.

I'm trying to perform CaImAn to my 2p-data with offline processing.
But the memory-mapping files during processing are too large and consuming the storage space; I'd like to remove them after the signal extraction except for the files I need when I recalculate.

For this, which file(s) is(are) necessary in the signal extraction?
My dataset(512px x 512px x 36000frames) is 5 separated tiffs; however, I performed processing based on the demo script, and then "6" mapped files will appear.
About these 6 memmaped files,
1, set of 5 memmap files; [basename]0000_[image dims][order]frames[total num frames/5],...,[basename]0004[image dims][order]frames[total num frames/5]
and
2: [basename][image dims]_[order]frames[total num of frames]
Total sizes of 1 and 2 are almost equal.
Are the files of 1 temporally used for creating file 2? In other words, so will the file 2 (maybe, a concatenated version of pieces of 5 memmap files) finally be utilized for the signal extraction?
If so, I’d like to keep only that for future reprocessing and delete the other files. In this way, I can load the essential memory-mapped file without re-creating a full set of memory-mapped files in future processing, for example, parameter optimization by try-and-error.
Or, if a way to achieve this exists, please let me know how to do it.

[EricThomson]

Thanks for the question: just to clarify, you are starting your analysis using load_movie_chain()?

Could you spell out the exact filenames, in particular the order parts they are very important (this could be helpful for people to see who stumble on this thread). : is the first set in F order and the second file in C order?

iirc typically lots of files are generated for motion correction in F order, and then a single file is generated for CNMF in C order, where one pixel must be read over the entire movie. This is the important file for source extraction, as discussed some in the docs here:
https://caiman.readthedocs.io/en/latest/Getting_Started.html#memory-mapping.

But sometimes things happen on the back end that surprise me, so let me know if I'm wrong. 😄

tldr: C order is the one for signal extraction, and is generated from the F-order memmapped file(s) that was created and used for motion correction. (and as an aside, the memmaped files are bigger than the originals -- which are often uint8 -- because they are recast as floats because we do lots of floating point math on the back end).

[masa6161]

Thanks for your reply!

just to clarify, you are starting your analysis using load_movie_chain()?

According to the demo-script, I didn't use load_movie_chain().
Firstly I've created memory mapped files with this code:

fname_mmap = cm.save_memmap(fnames, base_name=mmap_name.split("*")[0]+"_", order='C',
                                border_to_0=0, dview=cluster)

fnames is list of absolute path of tiff files.
mmap_name is defined by myself programmatically, and this is as below:
mmap_name = "mmap_240117_M2GNG#01-1" + "*" + str(num_frames) + ".mmap"
This is because to detect the concatenated file. num_frames is 36000 in this case.
Is it preferred that I have to use load_movie_chain() instead of this?

Could you spell out the exact filenames, in particular the order parts they are very important (this could be helpful for people to see who stumble on this thread). : is the first set in F order and the second file in C order?

The filenames of my dataset are below:
Reg_240117_M2GNG#01-1-N01.tif
Reg_240117_M2GNG#01-1-N02.tif
Reg_240117_M2GNG#01-1-N03.tif
Reg_240117_M2GNG#01-1-N04.tif
Reg_240117_M2GNG#01-1-N05.tif
And memmapped files are:
_mmap_240117_M2GNG#01-1-_0000_d1_512_d2_512_d3_1_order_C.mmap
mmap_240117_M2GNG#01-1-_0001_d1_512_d2_512_d3_1_order_C.mmap
mmap_240117_M2GNG#01-1-_0002_d1_512_d2_512_d3_1_order_C.mmap
mmap_240117_M2GNG#01-1-_0003_d1_512_d2_512_d3_1_order_C.mmap
mmap_240117_M2GNG#01-1-0004_d1_512_d2_512_d3_1_order_C.mmap
mmap_240117_M2GNG#01-1__d1_512_d2_512_d3_1_order_C_frames_36000.mmap
So all memmapped files seem in C-order.

tldr: C order is the one for signal extraction, and is generated from the F-order memmapped file(s) that was created and used for motion correction. (and as an aside, the memmaped files are bigger than the originals -- which are often uint8 -- because they are recast as floats because we do lots of floating point math on the back end).

As your explanations, if I separately perform the motion-correction before the signal extraction, each separated memmap files are utilized for creating a "concatenated" C-order file, right?
If so, it is no problem to remove separated files after concatenation. Actually, for now, the extraction is performed with this concatenated file directly and the processing might be okay.
Now, I used the code I wrote to remove the separated files, and if the final memmaped file exists in the specific directory, it is passed for signal extraction. The package has a built-in function to achieve this, please inform me!

And I'm slightly confusing why reshaping the images variable with "F-order" after reading-out the C-order memmapped file as;

Yr, dims, num_frames = cm.load_memmap(fname_mmap)
images = np.reshape(Yr.T, [num_frames] + list(dims), order='F') # load frames in python format (T x X x Y)

Probably this may be a basic question (because I've touched the Python environment from last week)...

[EricThomson]

If you have something that is working, just do what you are ok with, but as discussed in the demo (in the section
Working with multiple files we provide the method load_movie_chain() for loading/concatenating multiple files).

Very good question about that weird order='F' thing at the end. That is a trick that is there to just ensure that the images array is in the right shape, and doesn't have to do with memory mapping. If you remove it, the movie won't be in the correct Frames x Row x Column shape. This confused me for a long time when I first started working with Caiman (because it happens right when we are discussing F/C order for memory mapping 😬 ). (That comment in the code is from the days when it was ported from Matlab, where movies are stored in 'row x column x frame' format -- there are a lot of vestigial Matlab bits in there, echoes of indexing and shaping from days of yore).

[masa6161]

Thanks for your reply and explanations of a little (but very important) trick!

I see about the load_movie_chain(). I've now used the load_movie_chain(); it is very useful cause all tiffs are loaded simultaneously to RAM!
But in the demo_pipeline.ipynb, the data-loading to RAM is performed load_movie_chain(), however, the memory-mapping before the signal extraction runs with below code:

mc_memmapped_fname = cm.save_memmap(movie_path, base_name='memmap_',
                                     order='C', border_to_0=0, dview=cluster

By using this, irrespective if I've loaded data with load_movie_chain(), separated memmapped files and a concatenated one will be generated, right? I was confused when I first read the demo...

By the way, I noticed that the save_memmap() accepts not only a list of tiff files but also a list of arrays (it is documented in the function!). Does this mean that "a single memmap" can be obtained by passing data loaded from a series of tiffs with load_movie_chain() as like this?

images = [cm.load_movie_chain(movie_path)] # packing loaded movie as a list
fname_mmap = cm.save_memmap(images, base_name="mmap_", order='C', border_to_0=0, dview=None)

As a result of I've tried this snippet, I might be able to get ONLY a single concatenated memmap file! That's what I'd like to do!
Now, my modified pipeline using this snippet is running. I'll report the result of this later!

[masa6161]

Hi. I'm back to report the result of my experiment.
I could directly create a single memmapped file (without building-up from partial memmapped files) I'd shown in the previous post, and correctly compute signal extraction with this file!
It might be convenient for people who have enough RAM space to load the data for processing.
I hope this tips helps people like me.