Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix links to images in notebooks #1261

Merged
merged 1 commit into from
Feb 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 15 additions & 15 deletions demos/notebooks/demo_pipeline.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
"# CNMF demo pipeline: Intro\n",
"A full pipeline for the analysis of a two-photon calcium imaging dataset using the CaImAn software package. It demonstrates how to use Caiman for the following analysis steps:\n",
"\n",
"![full workflow cartoon](../../docs/img/full_cnmf_workflow.jpg)\n",
"![Full CNMF Workflow](https://raw.githubusercontent.com/flatironinstitute/CaImAn/main/docs/img/full_cnmf_workflow.jpg)\n",
"\n",
"1) Apply the nonrigid motion correction (NoRMCorre) algorithm for motion correction.\n",
"2) Apply the constrained nonnegative matrix factorization (CNMF) source separation algorithm to extract initial estimates of neuronal spatial footprints and calcium traces. \n",
Expand Down Expand Up @@ -197,9 +197,9 @@
" <h2>Displaying large files</h2>\n",
"Loading a movie with <em>cm.load()</em> pulls the data into memory, which is not always feasible. When working with your own data, you might need to adapt the above code when working with extremely large files. Caiman provides tools to handle this use case. One, you can just load some of your data into a movie object using the `subindices` argument to the `load()` function. For example, if you just want to load the first 500 frames of a movie, you can send it <em>subindices=np.arange(0,500)</em>. \n",
"\n",
"If you don't want to truncate your movie, there is a <em>play_movie()</em> function that behaves just like <em>movie.play()</em>, but it doesn't load the movie into memory. Rather, it takes the filename as an argument and loads frames from disk as they are shown. If you want to use it, just import using <em>from caiman.base.movies import play_movie</em> and read the documentation. We don't use it for this demo because the demo movie is small, and we do some calculations on the loaded movie array. \n",
"<p>If you don't want to truncate your movie, there is a <em>play_movie()</em> function that behaves just like <em>movie.play()</em>, but it doesn't load the movie into memory. Rather, it takes the filename as an argument and loads frames from disk as they are shown. If you want to use it, just import using <em>from caiman.base.movies import play_movie</em> and read the documentation. We don't use it for this demo because the demo movie is small, and we do some calculations on the loaded movie array.</p>\n",
"\n",
"Another option for viewing very large movies is to use the <a href=https://github.com/fastplotlib/fastplotlib>fastplotlib library</a>, which leverages the GPU to provide interactive visualization within Jupyter notebooks (we discuss this more below).\n",
"<p>Another option for viewing very large movies is to use the <a href=https://github.com/fastplotlib/fastplotlib>fastplotlib library</a>, which leverages the GPU to provide interactive visualization within Jupyter notebooks (we discuss this more below).</p>\n",
"</div>"
]
},
Expand Down Expand Up @@ -448,9 +448,9 @@
" <h2 style=\"margin-top: 0;\">Optimizing performance</h2> \n",
"If you hit memory issues later, there are a few things you can do. First, you may want to lower the number of processors you are using. Each processor uses more RAM, and on a workstation with many processors, you can sometimes get better performance by reducing <em>num_processors_to_use</em>.The best way to determine the optimal number is by trial and error. When you set <em>num_processors_to_use</em> variable to <em>None</em>, it defaults to <i>one</i> less than the total number of CPU cores available (the reason we don't automatically set it to the total number of cores is because in practice this typically leads to worse performance).\n",
"\n",
"Second, if your system has less than 32GB of RAM, and things are running slowly or you are running out of memory, then get more RAM. While you can sometimes get away with less, we recommend a *bare minimum* level of 16GB of RAM, but more is better. 32GB RAM is acceptable, 64GB or more is best. Obviously, this will depend on the size of your data sets. \n",
"<p>Second, if your system has less than 32GB of RAM, and things are running slowly or you are running out of memory, then get more RAM. While you can sometimes get away with less, we recommend a *bare minimum* level of 16GB of RAM, but more is better. 32GB RAM is acceptable, 64GB or more is best. Obviously, this will depend on the size of your data sets.</p>\n",
"\n",
"Third, if none of your memory optimizations work, you may just have too much data for offline CNMF. For this case, we also provide an online version of CNMF (OnACID), which uses a small number of frames to initialize the spatial and temporal components, and iteratively updates them with new data. This uses much less memory than the offline approach. The demo notebook for OnACID is found in <a href=\"./demo_OnACID_mesoscope.ipynb\">demo_OnACID_mesoscope.ipynb</a>. See the <a href=\"https://pubmed.ncbi.nlm.nih.gov/30652683/\">Caiman paper</a> for more discussion.\n",
"<p>Third, if none of your memory optimizations work, you may just have too much data for offline CNMF. For this case, we also provide an online version of CNMF (OnACID), which uses a small number of frames to initialize the spatial and temporal components, and iteratively updates them with new data. This uses much less memory than the offline approach. The demo notebook for OnACID is found in <a href=\"./demo_OnACID_mesoscope.ipynb\">demo_OnACID_mesoscope.ipynb</a>. See the <a href=\"https://pubmed.ncbi.nlm.nih.gov/30652683/\">Caiman paper</a> for more discussion.</p>\n",
"</div>"
]
},
Expand All @@ -461,7 +461,7 @@
"## Motion Correction\n",
"The first substantive step in our analysis pipeline is to remove motion artifacts from the original movie:\n",
"\n",
"<img src=\"../../docs/img/normcorre_workflow.jpg\" alt=\"motion correction workflow\" width=\"700\"/>\n",
"<img src=\"https://raw.githubusercontent.com/flatironinstitute/CaImAn/main/docs/img/normcorre_workflow.jpg\" alt=\"motion correction workflow\" width=\"700\"/>\n",
"\n",
"It is *very* important to get rid of motion artifacts, as the subsequent CNMF source separation algorithm assumes that each pixel represents the same region of space\n",
"\n",
Expand Down Expand Up @@ -658,7 +658,7 @@
"\n",
"Everything is now set up for running CNMF. This algorithm simultaneously extracts the *spatial footprint* and corresponding *calcium trace* for each component. \n",
"\n",
"![cnmf patch flow image](../../docs/img/cnmf_workflow.jpg)\n",
"![cnmf patch flow image](https://raw.githubusercontent.com/flatironinstitute/CaImAn/main/docs/img/cnmf_workflow.jpg)\n",
"\n",
"It also performs *deconvolution*, providing an estimate of the spike count that generated the calcium signal in the movie. "
]
Expand All @@ -669,7 +669,7 @@
"source": [
"The algorithm is parallelized as illustrated here:\n",
"\n",
"<img src=\"../../docs/img/cnmf_patches.jpg\" alt=\"cnmf patch flow\" width=\"500\"/>\n",
"<img src=\"https://raw.githubusercontent.com/flatironinstitute/CaImAn/main/docs/img/cnmf_patches.jpg\" alt=\"cnmf patch flow\" width=\"500\"/>\n",
"\n",
"1) The movie field of view is split into overlapping patches.\n",
"2) These patches are processed in parallel by the CNMF algorithm. The degree of parallelization depends on your available computing power: if you have just one CPU then the patches will be processed sequentially. \n",
Expand Down Expand Up @@ -919,7 +919,7 @@
" <h2 style=\"margin-top: 0;\">More on Estimates</h2> \n",
"The estimates object contains a great deal of information. The attributes are discussed in more detail <a href=\"https://caiman.readthedocs.io/en/latest/Getting_Started.html#result-interpretation\">in the documentation</a>, but you might also find exploring the <a href=\"https://github.com/flatironinstitute/CaImAn/blob/main/caiman/source_extraction/cnmf/estimates.py\">source code</a> helpful. For instance, while most users initially care about the extracted calcium signals <em>C</em> and spatial footprints <em>A</em>, the <b>background model</b> is also very important. The background model is included in the estimate in fields <em>b</em> and <em>f</em> (which correspond to the spatial and temporal components of the low-rank background model, respectively). We discuss the background model more below.\n",
"\n",
"Also, we realize that attribute names like <em>A</em> are not very informative or Pythonic. These names are rooted in mathematical conventions from the original papers in the literature.\n",
"<p>We realize that attribute names like <em>A</em> are not very informative or Pythonic. These names are rooted in mathematical conventions from the original papers in the literature.</p>\n",
"</div>"
]
},
Expand All @@ -930,7 +930,7 @@
"# Component Evaluation\n",
"As already mentioned, the initial estimates produced by CNMF contains many spurious components. Our next step is to do some some quality control, cutting out the bad estimates to arrive at our final set of estimates:\n",
"\n",
"![component evaluation image](../../docs/img/evaluation_workflow.jpg)"
"![component evaluation image](https://raw.githubusercontent.com/flatironinstitute/CaImAn/main/docs/img/evaluation_workflow.jpg)"
]
},
{
Expand All @@ -945,7 +945,7 @@
"\n",
"The first two criteria are illustrated schematically here (see also Figure 2 of <a href=\"https://elifesciences.org/articles/38173\">the Caiman paper</a>):\n",
"\n",
"![component evaluation image](../../docs/img/component_evaluation.jpg)"
"![component evaluation image](https://raw.githubusercontent.com/flatironinstitute/CaImAn/main/docs/img/component_evaluation.jpg)"
]
},
{
Expand Down Expand Up @@ -1008,7 +1008,7 @@
"In practice, SNR is the most important evaluation factor. The spatial correlation factors are less important. In particular, the CNN for spatial evaluation may be inaccurate if your neural components are not \"canonically\" shaped somata. \n",
"\n",
"\n",
"You may have noticed from the description above, that when running <em>evaluate_components()</em> the three evaluation thresholds are appied <em>inclusively</em>: if a component is above <em>any</em> of the thresholds, it will pass muster. This was found in practice to be reasonable (e.g., a low SNR component that is very strongly neuronally shaped tends to not be an accident: it is just a very low SNR neuron). However, there is a second set of <b>absolute</b> threshold parameters set for each criterion. If a component is <em>below</em> this absolute threshold for any of the evaluation parameters, it will be discarded: these are the <em>SNR_lowest</em>, <em>rval_lowest</em>, and <em>cnn_lowest</em>, respectively. \n",
"<p>When running <em>evaluate_components()</em> the three evaluation thresholds are appied <em>inclusively</em>: if a component is above <em>any</em> of the thresholds, it will pass muster. This was found in practice to be reasonable (e.g., a low SNR component that is very strongly neuronally shaped tends to not be an accident: it is just a very low SNR neuron). However, there is a second set of <b>absolute</b> threshold parameters set for each criterion. If a component is <em>below</em> this absolute threshold for any of the evaluation parameters, it will be discarded: these are the <em>SNR_lowest</em>, <em>rval_lowest</em>, and <em>cnn_lowest</em>, respectively.</p> \n",
"</div>"
]
},
Expand Down Expand Up @@ -1206,13 +1206,13 @@
"metadata": {},
"source": [
"## Extract $\\Delta F/F$ values\n",
"So far our calcium traces are in arbitrary units. It is common to report the calcium fluorescence relative to some baseline value $F_0$:\n",
"So far our calcium traces are in arbitrary units. It is more common to report the calcium fluorescence relative to some baseline value $F_0$:\n",
"\n",
"$$\n",
"\\Delta F/F = (F(t)-F_0)/F_0\n",
"$$\n",
"\n",
"Traditionally, the baseline value was calculated during some initial period of queiscience (e.g., in the visual system, when the animal was sitting in the dark). However, it is problematic to make any assumptions about a \"quiet\" baseline period, so researchers have started to use a moving percentile for $F_0$. This is also what Caiman does.\n",
"Traditionally, the baseline value was calculated during some initial period of queiscience (e.g., in the visual system, when the animal was sitting in the dark). However, it is problematic to make any assumptions about a \"quiet\" baseline period, so researchers have started to use a moving average to calculate $F_0$. This is also what Caiman does.\n",
"\n",
"More specifically, in Caiman the baseline is a running percentile calculated over a `frames_window` moving window. You can calculate $\\Delta F/F$ using raw traces or the denoised traces in C (this is toggled using the `use_residuals` argument):"
]
Expand Down Expand Up @@ -1245,7 +1245,7 @@
"source": [
"<div class=\"alert alert-info\" markdown=\"1\">\n",
" <h2 style=\"margin-top: 0;\">More on $\\Delta F/F$</h2> \n",
"The above discussion of dff leaves out some details about it is actually calculated in Caiman. Behind the scenes, Caiman projects the background activity (see the discussion on the background model below) to the spatial footprint of the neuron and adds a moving percentile of this projected trace to the denominator. This acts as an empirical fudge factor that can keep results from getting too large when very low values for $F_0$ are in the denominator (especially when using denoised traces). Users may prefer to use the more traditional equation directly. Thanks to Peter Rupprecht for a helpful discussion of this topic. \n",
"The above discussion of dff leaves out some details about it is actually calculated in Caiman. Behind the scenes, Caiman projects the background activity to the spatial footprint of the neuron and adds a moving percentile of this projected trace to the denominator (see the discussion on the background model below). This acts as an empirical fudge factor that can keep results from getting too large when very low values for $F_0$ are in the denominator (especially when using denoised traces). Users may prefer to use the more traditional equation directly. Thanks to Peter Rupprecht for a helpful discussion of this topic. \n",
"</div>"
]
},
Expand Down
Loading
Loading