IncompleteRead Error while trying to load demo videos

[raymondwjang]

1. Operating System (Linux, MacOS, Windows): MacOS
2. Hardware type (x86, ARM..) and RAM: Apple Silicon M1
3. Python Version (e.g. 3.9): 3.11
4. Caiman version (e.g. 1.9.12): 1.11.2
5. Which demo exhibits the problem (if applicable): demo_online_cnmfE.ipynb, demo_realtime_cnmfE.ipynb
6. How you installed Caiman (pure conda, conda + compile, colab, ..):

conda install -n base -c conda-forge mamba   # install mamba in base environment
mamba create -n caiman -c conda-forge caiman # install caiman
conda activate caiman  # activate virtual environment
caimanmanager install

9. Details:
   Downloading demo videos fails in demo_online_cnmfE.ipynb, and demo_realtime_cnmfE.ipynb notebooks. They throw identical errors.

within demo_realtime_cnmfE.ipynb,

iterator = get_iterator(download_demo('blood_vessel_10Hz.mat'))

m = cm.movie(np.array([next(iterator) for t in range(init_batch)], dtype='float32'))

throws:

IncompleteRead                            Traceback (most recent call last)
Cell In[5], line 5
      2 T = 6000           # total number of frames 
      3 fr = 10            # frame rate (Hz)
----> 5 iterator = get_iterator(download_demo('blood_vessel_10Hz.mat'))
      7 m = cm.movie(np.array([next(iterator) for t in range(init_batch)], dtype='float32'))

File /opt/anaconda3/envs/caiman/lib/python3.11/site-packages/caiman/utils/utils.py:111, in download_demo(name, save_folder)
    108     req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
    109     f = urlopen(req, context=urllib_context)
--> 111 data = f.read()
    112 with open(path_movie, "wb") as code:
    113     code.write(data)

File /opt/anaconda3/envs/caiman/lib/python3.11/http/client.py:489, in HTTPResponse.read(self, amt)
    487 else:
    488     try:
--> 489         s = self._safe_read(self.length)
    490     except IncompleteRead:
    491         self._close_conn()

File /opt/anaconda3/envs/caiman/lib/python3.11/http/client.py:640, in HTTPResponse._safe_read(self, amt)
    638 data = self.fp.read(amt)
    639 if len(data) < amt:
--> 640     raise IncompleteRead(data, amt-len(data))
    641 return data

IncompleteRead: IncompleteRead(292606688 bytes read, 239976515 more expected)

and within demo_online_cnmfE.ipynb,

movie_ind = 0   # 0 for avi, 1 for the tif (in case avi loading troubles)
fnames = ['msCam13.avi', 'data_endoscope.tif']  # filename to be processed
fnames = [download_demo(fnames[movie_ind])] 

throws a similar error.

The read/expected byte size change every trial. While demo_online_cnmfE.ipynb can bypass the error by fetching the data_endoscope.tif file instead (changing movie_ind to 1.), there is no similar option for demo_realtime_cnmfE.ipynb

[pgunn]

I'll start looking into this locally, although I thought I had taested this earlier; is there any chance your HD may have filled up resulting in corrupt files, or anything like that?

[raymondwjang]

There's about 100 GB of storage remaining, so I doubt that's the issue. It also fails pretty fast - within a minute.

[pgunn]

Ok, I'll see what I can figure out then; I may have followup questions (e.g. opencv version) depending on how that goes.

[pgunn]

Having tried to reproduce this and having looked more carefully at the error,

1. It's failing in the http library, not anyplace particularly interesting in caiman's code
2. It doesn't reproduce locally on Linux

So, we have a few routes forward to see if we can resolve this; my two main theories are that either:

A) You're somplace with unusual network issues, or
B) Something unusual is happening with the https certs

This chunk of code shows where the download_demo() function retrieves demo files from:

https://github.com/flatironinstitute/CaImAn/blob/main/caiman/utils/utils.py#L66

If you go into ~/caiman_data/example_movies/ and remove the bad file and then do a
wget https://caiman.flatironinstitute.org/~neuro/caiman_downloadables/blood_vessel_10Hz.mat

(adjust to use curl if you don't have a wget)
Do you get a file that works?

This should help us figure out what's going on (you may also see, if you're on a laptop if you get the same issue while running on another network, e.g. at home or in a coffeeshop)

[raymondwjang]

Sorry about the delay @pgunn.

yes, running wget https://caiman.flatironinstitute.org/~neuro/caiman_downloadables/blood_vessel_10Hz.mat does indeed work. I actually had already tried running the same code on multiple wifis (home wifi, office wifi, etc.), and they all failed with the same error, unfortunately.

[raymondwjang]

I have just successfully run

wget https://caiman.flatironinstitute.org/~neuro/caiman_downloadables/msCam13.avi

to download the same video file that the demo_online_cnmfE.ipynb attempts to download and fails. Just to double check, I've also confirmed that the notebook download run fails on the same wifi, simultaneously.

[pgunn]

This is puzzling; for now you can work around this with manual fetches of the demo data; it may be useful to see if you get the same results from the CLI demos.

Occasionally in the past we saw issues with certs in python, but that usually would cause an upfront failure rather than a failure partway through a download.

I'd expect that if a router might cancel big downloads, it'd do it the same for wget vs python doing the same thing. Unless perhaps it's different http headers leading to different treatment. It's also possible OSX itself is being weird, although that seems unlikely.

Knowing you have a pretty easy workaround, it may not be worth your time to debug this the rest of the way, but you have me curious and I'm happy to do it - up to you. Have you tried doing this from some different network? e.g. a coffeeshop, home, or some other place where you've seen this issue?

[raymondwjang]

Yeah, those were my expectations, as well. I haven't tried wget from different networks, but I have tried and confirmed that the download_demos function fails with the same error trace across different networks. (home, school, office)

As far as I'm concerned, it does not actually impact my workflow anymore since you have identified a workaround for me :) I am happy to pair with debugging, if you'd like to RCA.