CNMF notebook update #1075

EricThomson · 2023-05-03T14:43:32Z

Description

Updating CNMF demo notebook. This isn't a fix, but part of roadmap to revitalize all the major notebooks to include more explanation and generally increasing accessibility to the code base. This is part of:

Changes in documentation or new examples for demos and/or use cases.

Aims

This is a work in progress, will add more here as I work through it, but main goals for now:

pgunn · 2023-05-03T15:03:22Z

demos/notebooks/demo_pipeline.ipynb

@@ -4,7 +4,8 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "<html><head><meta content=\"text/html; charset=UTF-8\" http-equiv=\"content-type\"><style type=\"text/css\">ol</style></head><body class=\"c5\"><p class=\"c0 c4\"><span class=\"c3\"></span></p><p class=\"c2 title\" id=\"h.rrbabt268i6e\"><h1>CaImAn&rsquo;s Demo pipeline</h1></p><p class=\"c0\"><span class=\"c3\">This notebook will help to demonstrate the process of CaImAn and how it uses different functions to denoise, deconvolve and demix neurons from a two-photon Calcium Imaging dataset. The demo shows how to construct the `params`, `MotionCorrect` and `cnmf` objects and call the relevant functions. You can also run a large part of the pipeline with a single method (`cnmf.fit_file`). See inside for details.\n",
+    "# CNMF demo pipeline\n",
+    "This notebook will help to demonstrate the process of CaImAn and how it uses different functions to denoise, deconvolve and demix neurons from a two-photon Calcium Imaging dataset. The demo shows how to construct the `params`, `MotionCorrect` and `cnmf` objects and call the relevant functions. You can also run a large part of the pipeline with a single method (`cnmf.fit_file`). See inside for details.\n",


We should probably change the wording to avoid the potentially confusing future tense here.

"This notebook demonstrates the process of using Caiman and how it uses different functions..."

kushalkolar · 2023-05-04T02:53:49Z

Thoughts:

Explain component evaluation/removal of false positives

This might be better done in a different nb entirely. I will eventually make a visualization for this in the mesmerize demo nbs, but it just uses the Estimates so it's easy to adapt for a nb in the caiman repo if you want. There are quite a few features that can be exposed better, such as merging components.

Deconvolution parameters and model (I may need some help with this)
How to run deconvolution after running cnmf to explore parameters.

Might be more organized in a separate nb?

DFoF calculation explain

Also a dedicate notebook with a longer discussion on this would be really useful. We could use some of the ideas from Peter Rupretch too.

EricThomson · 2023-05-04T03:14:45Z

Thoughts:

Explain component evaluation/removal of false positives

This might be better done in a different nb entirely. I will eventually make a visualization for this in the mesmerize demo nbs, but it just uses the Estimates so it's easy to adapt for a nb in the caiman repo if you want. There are quite a few features that can be exposed better, such as merging components.

Thanks Kushal yes probably shouldn't go down the rabbit hole with this, but was just thinking of adding a tad more explanation for the component evaluation step than is already there. Should probably be a separate notebook for a deep dive (that could easily be a very long notebook).

Deconvolution parameters and model (I may need some help with this)
How to run deconvolution after running cnmf to explore parameters.

Might be more organized in a separate nb?

Yes definitely I shouldn't aim to go deep with any of this the nb would take too long to make.

DFoF calculation explain

Also a dedicate notebook with a longer discussion on this would be really useful. We could use some of the ideas from Peter Rupretch too.

Yes, as before I should clarify I don't want to go deep, just a bit more than is done now (which is effectively nothing). I just remember finishing the notebooks and being like...wait did deconvolution happen yet? I'd like it to give an accurate gloss that makes users feel like they've got the basic pipeline explained, but leave the details to "auxilliary" notebooks.

pgunn

some phrasing/typo things

pgunn · 2023-05-09T15:05:13Z

demos/notebooks/demo_pipeline.ipynb

@@ -187,11 +187,9 @@
   "metadata": {},
   "source": [
    "## Setting up parameters\n",
-    "While there are many parameters used in CaImAn for the different steps in the pipeline, you only have to think about a few of them in each step, which we will discuss below. \n",
+    "Parameters are defined in a standalone parameters object. We create this object by passing parameters as a single dictionary to the `CNMFParams` class below. The parameters *not* explicitly defined in the dictionary will assume default values. A paramter object (let's call it `parameters`) is effectively a collection of dictionaries that each contains parameters relevant to different settings. Some related to the dataset in general (`parameters.data`), while most are relaed to particular aspects of the workflow such as motion correction (`parameters.motion`), component extraction (`parameters.spatial` and `parameters.temporal`), quality evaluation (`parameters.quality`), and others. \n",


Typo: "A parameter object"

pgunn · 2023-05-09T15:05:30Z

demos/notebooks/demo_pipeline.ipynb

@@ -187,11 +187,9 @@
   "metadata": {},
   "source": [
    "## Setting up parameters\n",
-    "While there are many parameters used in CaImAn for the different steps in the pipeline, you only have to think about a few of them in each step, which we will discuss below. \n",
+    "Parameters are defined in a standalone parameters object. We create this object by passing parameters as a single dictionary to the `CNMFParams` class below. The parameters *not* explicitly defined in the dictionary will assume default values. A paramter object (let's call it `parameters`) is effectively a collection of dictionaries that each contains parameters relevant to different settings. Some related to the dataset in general (`parameters.data`), while most are relaed to particular aspects of the workflow such as motion correction (`parameters.motion`), component extraction (`parameters.spatial` and `parameters.temporal`), quality evaluation (`parameters.quality`), and others. \n",


"that each contains" -> "each containing"

pgunn · 2023-05-09T15:05:49Z

demos/notebooks/demo_pipeline.ipynb

@@ -187,11 +187,9 @@
   "metadata": {},
   "source": [
    "## Setting up parameters\n",
-    "While there are many parameters used in CaImAn for the different steps in the pipeline, you only have to think about a few of them in each step, which we will discuss below. \n",
+    "Parameters are defined in a standalone parameters object. We create this object by passing parameters as a single dictionary to the `CNMFParams` class below. The parameters *not* explicitly defined in the dictionary will assume default values. A paramter object (let's call it `parameters`) is effectively a collection of dictionaries that each contains parameters relevant to different settings. Some related to the dataset in general (`parameters.data`), while most are relaed to particular aspects of the workflow such as motion correction (`parameters.motion`), component extraction (`parameters.spatial` and `parameters.temporal`), quality evaluation (`parameters.quality`), and others. \n",


"Some are related"

pgunn · 2023-05-09T15:06:06Z

demos/notebooks/demo_pipeline.ipynb

@@ -187,11 +187,9 @@
   "metadata": {},
   "source": [
    "## Setting up parameters\n",
-    "While there are many parameters used in CaImAn for the different steps in the pipeline, you only have to think about a few of them in each step, which we will discuss below. \n",
+    "Parameters are defined in a standalone parameters object. We create this object by passing parameters as a single dictionary to the `CNMFParams` class below. The parameters *not* explicitly defined in the dictionary will assume default values. A paramter object (let's call it `parameters`) is effectively a collection of dictionaries that each contains parameters relevant to different settings. Some related to the dataset in general (`parameters.data`), while most are relaed to particular aspects of the workflow such as motion correction (`parameters.motion`), component extraction (`parameters.spatial` and `parameters.temporal`), quality evaluation (`parameters.quality`), and others. \n",


typo: "while most are related"

Thanks Pat, changes made!

demos/notebooks/demo_pipeline.ipynb

pgunn · 2023-05-12T18:19:19Z

demos/notebooks/demo_pipeline.ipynb

+    "## Select files for processing\n",
+    "Many acquisition systems break up data from a single session across multiple files. This demo shows how to work with lists of filepaths that represent multiple movies from the same recording session. While this demo works with `tif` files, Caiman can handle movies in multiple common formats such as:\n",
+    "\n",
+    "    tiff, hdf5, nwb, avi, zarr, h5, npz, n5\n",


hdf5 and h5 should not both be in the list.

I changed it to hdf5/h5

EricThomson · 2023-05-17T15:38:15Z

Realized I had branched from main not dev, so fixed that. No need for re-review all these commits they are repeats. :)

cleared outputs minor tweak to CNMF intro

EricThomson · 2023-10-22T18:35:29Z

Note not quite done -- need to update links (it still mentions appendices in a couple of places), and I want to decrease verbosity in a couple of places.

EricThomson · 2023-10-24T17:22:48Z

Will look at the merge conflict. Not sure what is up with that.

EricThomson · 2023-10-24T18:25:28Z

Merge conflicts handled, review comments absorbed.

kushalkolar · 2024-01-20T04:35:47Z

caiman/cluster.py

+        backend
+            One of:
+                'multiprocessing' - Use multiprocessing library
+                'ipyparallel' - Use ipyparallel instead (better on Windows?)


interesting, is this related to the requirement to have an if __name__ == "__main__" on windows?

demos/notebooks/demo_pipeline.ipynb

…/CaImAn into cnmf_notebook_update

EricThomson · 2024-01-22T18:58:46Z

In interests of letting good be the enemy of the perfect, and letting user feedback guide development, I think we should merge into dev unless there are major problems. I see this as useful step forward for initial users over previous iteration. Could add stuff on deconvolution, but I don't understand it very well so I'm not going to try right now.

Plan to go through CNMFE this week with similar course-toothed comb.

EricThomson added the status: in progress label May 3, 2023

pgunn reviewed May 3, 2023

View reviewed changes

EricThomson marked this pull request as draft May 3, 2023 17:08

pgunn reviewed May 9, 2023

View reviewed changes

pgunn reviewed May 12, 2023

View reviewed changes

demos/notebooks/demo_pipeline.ipynb Show resolved Hide resolved

pgunn reviewed May 12, 2023

View reviewed changes

EricThomson added 15 commits May 17, 2023 11:13

starting work updating cnmf demo

aa98b56

updated intro section

e7fbc74

updated logging section

f35aed2

updated logger explanation

b886161

updated file selection section

76d265e

added bit about multisession to file loading session

40e0030

path fixing

8ea1aec

movie viewer discussion added

0d1e821

updated cluster setup section

6c73b34

revised cluster setup description

4724dd5

added temp placeholder image of workflow

e7f3d8b

added subindices discussion to movie player

5ef769d

updated intro to parameter object

52309c9

edited parameter description

d7cb2f9

rebased to dev from master oops

a15d837

EricThomson force-pushed the cnmf_notebook_update branch from 99dd986 to a15d837 Compare May 17, 2023 15:36

EricThomson added 5 commits May 18, 2023 11:12

updated handling of logging

82a1eda

simplified download and added movie type info highlight

4a486d3

simplified movie loading/viewing sections

a711340

updated through cluster

0b6351a

motion correction section updated

098e3d4

cleared outputs minor tweak to CNMF intro

EricThomson added 5 commits October 22, 2023 13:44

removing appendices -- moved logging to Getting_Started.rst

55602a8

moved api discussion to docs

45260a1

cut cluster setup appendix, moved to docs

e9e2c32

cut memmapping appendix, moved to docs

d1fba38

appendices all moved to sidebars and deleted

a9c1b4b

updated based on review, appendices to docs, etc

4e46230

Merge branch 'dev' into cnmf_notebook_update

ab91335

EricThomson added 3 commits October 26, 2023 11:00

minor typos and word-choice changes

f07c7f9

removing excess verbiage

93cc1f6

cut and simplified notebook

cc4b31b

EricThomson mentioned this pull request Jan 7, 2024

CaImAn 2023 Roadmap #1142

Open

45 tasks

EricThomson added 4 commits January 18, 2024 17:36

preliminary save traces to csv added

5a7e15b

explain how to save results as csv

5fb4ab7

Merge branch 'dev' into cnmf_notebook_update

30c9db2

added explanation of dfof

70835d7

kushalkolar reviewed Jan 20, 2024

View reviewed changes

demos/notebooks/demo_pipeline.ipynb Show resolved Hide resolved

EricThomson added 4 commits January 22, 2024 13:04

updated discussion of dff

6a1a813

Merge branch 'cnmf_notebook_update' of https://github.com/EricThomson…

e2756eb

…/CaImAn into cnmf_notebook_update

added caveat about time stamps

161e516

gsig/gsiz just define what they are

fbf1418

EricThomson merged commit 50e99e7 into flatironinstitute:dev Jan 25, 2024
3 checks passed

This was referenced Jan 25, 2024

estimates.detrend_df_f use_residuals keyword has no effect #1188

Closed

Clarify gSiz parameter role(s) #1139

Open

Explain how to run CNMF/CNMFE on file that is already motion corrected, not memmapped #1083

Closed

EricThomson mentioned this pull request Mar 20, 2024

better documentation of serializing CNMF models to disk and loading them in #1141

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CNMF notebook update #1075

CNMF notebook update #1075

EricThomson commented May 3, 2023 •

edited

Loading

pgunn May 3, 2023

kushalkolar commented May 4, 2023

EricThomson commented May 4, 2023 •

edited

Loading

pgunn left a comment

pgunn May 9, 2023

pgunn May 9, 2023

pgunn May 9, 2023

pgunn May 9, 2023

EricThomson May 9, 2023

pgunn May 12, 2023

EricThomson Oct 17, 2023 •

edited

Loading

EricThomson commented May 17, 2023

EricThomson commented Oct 22, 2023

EricThomson commented Oct 24, 2023

EricThomson commented Oct 24, 2023

kushalkolar Jan 20, 2024

EricThomson commented Jan 22, 2024 •

edited

Loading

CNMF notebook update #1075

CNMF notebook update #1075

Conversation

EricThomson commented May 3, 2023 • edited Loading

Description

Aims

Choose a reason for hiding this comment

kushalkolar commented May 4, 2023

EricThomson commented May 4, 2023 • edited Loading

pgunn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EricThomson Oct 17, 2023 • edited Loading

Choose a reason for hiding this comment

EricThomson commented May 17, 2023

EricThomson commented Oct 22, 2023

EricThomson commented Oct 24, 2023

EricThomson commented Oct 24, 2023

Choose a reason for hiding this comment

EricThomson commented Jan 22, 2024 • edited Loading

EricThomson commented May 3, 2023 •

edited

Loading

EricThomson commented May 4, 2023 •

edited

Loading

EricThomson Oct 17, 2023 •

edited

Loading

EricThomson commented Jan 22, 2024 •

edited

Loading