Timelapse support

[lazigu]

Hi @Cryaaa, I need your help with one thing, maybe you can pinpoint the issue faster. The issue might have something to do with the code for the visualization of the cluster image, but it also might be somewhere deeper.
After seeing that video yesterday I was playing around with timelapses on this branch, and so far what works:

• Getting measurements for a timelapse. It loops through the whole timelapse and gets reg props data for each time point and in the end concatenates all (works for now just for area & intensity measurements, not with neighborhood data). There is also a "timepoint" column, which we might rename to "frame" to be consistent with Trackmate results.
• Then you can plot timepoint against the label, and nicely see if visualization works correctly (6th cell divides at 6th timepoint, so we get one more label, and it's good for looking for bugs):
  

It works well for 2-6 labels, but not 1st and 7th. This is what it displays if the 1st label is selected with a picker tool:

I was hoping we could discuss this next week in one of the meetings when we are both there, so I am just putting this here so if you wanted you could already take a look. Tell me if you want to send me this segmented dataset, it's 4D but I am using just 16 timepoints, so it's great to try out stuff because it's not computationally demanding.

[Cryaaa]

Hey @lazigu!
Definitely send the dataset (we transfer link or whatever you use usually would be best) so I can try this myself. I will look into this a bit, maybe we can discuss this issue on Tuesday next week in detail?

[lazigu]

@Cryaaa good, Tuesday fits! I sent you the data.

[Cryaaa]

@lazigu just a small update. The problem seems to lie somewhere deep (not just with the generation of the cluster image). What happens is that the label column is displayed properly in the table but the connection to the labels in the image is wrong. For example the cell in the image with label = 1 gets given label 2 from the table and this shift is throughout the whole table. I'll see if I can track down where this issue is caused or we'll brainstorm tomorrow if I can't find anything today..

[Cryaaa]

and furthermore this issue also exists for non-timelapse images on the main branch which is worrying so it should have quite a high priority for solving..

[lazigu]

@Cryaaa I merged PR from Robert today #59, if you mean by the connection of reg props and the image when looking what is displayed in napari status bar, it should have been fixed. Just this branch should be also updated now from main. Is this what you meant?

[lazigu]

Forgot to mention, for the fix to work napari skimage regionprops newest version should be installed in the environment.

[Cryaaa]

@lazigu that might be it then! I'll give it a try where the Roberts fix is and that might be it already. I'm also very sure that exactly that bug Robert fixed is causing the issues at the start of this discussion..

[lazigu]

@Cryaaa I don’t think so that it is related to this issue :/ It fixes what is displayed in napari status bar, but actually nowhere else we are using the index column.

[Cryaaa]

@lazigu I think you're right there. I found that if there is a background label 0 with properties of the background that the problem we are having might be solved. The thing is then we always have the background in the table (even multiple backgrounds for timelapse images). But maybe Robert @haesleinhuepf knows a way to remove or ignore the background labels in the napari skimage regionprops table? I think we should discuss the rest in person tomorrow though, it will make explanations much easier.

[Cryaaa]

@lazigu, nevermind, I found the problem and it lied within the plotter widget. The next commit to this branch should fix it!

[lazigu]

• Discuss how to handle background label/row in layer.properties/reg props table Errors in the dimensionality reduction widget if there are NaN values in the regprops table #56.
• Should it be standardized that uploaded reg props from the file need to contain timepoint/frame column? Mention this in the documentation that it should.
• Settle on the name for the timepoint/frame column.
• Saving manual clustering results in the reg props table.
• Implement that the size of data points in the plot changes depending on how many data points are displayed.
• napari status bar displays correct "label", but not correct measurements (e.g. "area") because it shows measurements of the first timepoint and not the currently selected one. Should this issue be raised with napari developers?
• Neighbourhood Measurements for timelapse data
• Reverting back to the generation of a cluster image with clesperanto functions
• index column not added when files are uploaded making the napari status bar display the wrong label
• Highlighting of datapoints in current time frame (implemented but needs to be fine tuned for large dataclouds)

@Cryaaa feel free to add points to this mini-list!

[lazigu]

I moved this thread to the "Discussions" section, I think it is more relevant here.

[Cryaaa]

Hey @lazigu! Great idea to move the discussion here! I just ran into a new bug when processing large amounts of data. When performing timelapse measurement, the neighbourhood data is calculated after concatenating the regionprops of each timepoint meaning that the neighbourhood data is not calculated per timepoint but for the whole (sometimes very big) table for all timepoints. This causes napari to get stuck (and would cause results which aren't true). I will try and find a solution for it today though, this is just a heads up!

[lazigu]

@Cryaaa I didn't do timelapse support for neighborhood measurements yet, I think I mentioned it in the first post here :) So this should also go to the to do list.

[Cryaaa]

@lazigu, in that case I'll take care of it!

[Cryaaa]

@lazigu, I did some restructuring of the functions in ._measure in the latest commit to the branch. I have left the old function so you can see what I did differently. The main difference is: always working with pandas Dataframes when dealing with regionprops. I did not like that regionprops were sometimes a dict and sometimes a dataframe so I made it that they are always dataframes. I'm open todiscussing the changes I'm suggesting but we'll do that once you are back from Paris :)

[Cryaaa]

Hey this is another big point to add to the list of problems when working with timelapse data. The visualisation using the colormaps will not always work unfortunately since in some cases the points selected in the plotter widget will not be found in every time frame, which is why we might need to go back to generating a cluster image with the clesperanto functions. Since I still have the function for that functionality from my master thesis I will try and implement it in the next few days and see how it affects performance.

[Cryaaa]

The spot size now dynamically changes with number of points between some arbitrarily set boundaries. We may need to modiy the behaviour if we find use cases where it could be improved but for now its really cool to be able to visualise these really large UMAPs!

[Cryaaa]

@lazigu
I also did some tests to why the neighborhood measurements are failing for some time points and I cannot recreate the behaviour in napari, which is very strange. I have some notebooks where I calculate the region properties exactly like in the plugin (using a worker, using the same functions and also using the layers.data method for the images and even using the set features function) but I don't get any columns with nan values. I find this behaviour really really strange and don't really know how to interpret it. Maybe you can also run the notebook and see if it is the same for you? I have added it to the docs in the timelapse branch

[lazigu]

@Cryaaa
I´m also playing around with this issue, so I will keep you posted if I figure something out. I will also take a look at those notebooks, thanks!

[lazigu]

@Cryaaa very weird indeed, I am also not getting any NaN values in the notebook 🤔

[Cryaaa]

@lazigu very strange indeed, I don't even really know how to tackle the issue at the moment....

[haesleinhuepf]

Hi Laura and Ryan @lazigu and @Cryaaa ,

just to let you know, I recently updated napari-skimage-regionprops and napari-simpleitk-image-processing to make them support timelapse-data when doing quantiative measurements. We could use the same strategy in the clusters plotter, if we decide to refactor the measurement code. Precondition would be that we drop the custom widget and use a plain function and magicgui instead. We could then do this:

def measure(image, labels, ...):
     ....

measure_all_frames = analyze_all_frames(measure)
register_function(measure_all_frames, menu="Measurement > Measure of all frames (ncp)")

This might mean that we have to make the "upload CSV" a different, indepent function. And we could move it to napari-skimage-regionprops.

That's all just an idea. I'd be happy to actively contribute this. Let me know what you think and maybe we discuss in a meeting. I just wanted to avoid that we duplicate work in this context. ;-)

[Cryaaa]

@haesleinhuepf
since we are havng some troubles implementing the existing function it might be a good idea to do it this way! I definitely agree with avoiding duplication and I guess implementation of measurement is also always quite project specific. We should definitely discuss this in a meeting asap!

[lazigu]

Hi Robert @haesleinhuepf ! We are actually almost done with making measuring widget support timelapses, so I guess we have already duplicated a bit of work but it´s totally fine :) We were having problems just with the neighbourhood measurements, and if I remember it right napari-skimage-regionprops doesn´t have such measurements, or does it now? Overall, I like your idea, it would make clusters plotter code cleaner, since measure widget is so similar to napari-skimage-regionprops, and the plugin already depends on it. But I am afraid it might be way more work unless we really can´t solve this issue with neighbourhood measurements..
Maybe we can discuss this in a meeting on Thursday because I can´t join tomorrow?

[haesleinhuepf]

Yes, let's discuss Thursday 11 am 🙌

[Cryaaa]

@lazigu
we can discuss tomorrow in detaiil but I have found what was causing the problems with neighborhood measurements! The latest commit to the branch fixes it.

[Cryaaa]

The commit just now (07.04.22 16:08) implements the highlighting of current time slice datapoints into the time series branch. The code for visualisation specific parameters is now moved into its own file so that future changes to specifics can be easily implemented without changing the plotter widget too much.

[Cryaaa]

@lazigu,
Since we solved all of the issues I will make a PR and ask for some feedback in the BiAPOL group to see if some other issues appear :)