online_iNMF_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Iterative single-cell multi-omic integration using online learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Please load following modules\n",
    "import pyliger\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scenario 1: sampling minibatches from fully observed datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first create a Liger object by passing the filenames of HDF5 files containing the raw count data. The data can be downloaded [here](https://www.dropbox.com/sh/d7fnebmjfv9ueqi/AAA9vlwlCqe-P6T_iPYDier9a?dl=0). Liger assumes by default that the HDF5 files are formatted by the 10X CellRanger pipeline. Large datasets are often generated over multiple 10X runs (for example, multiple biological replicates). In such cases it may be necessary to merge the HDF5 files from each run into a single HDF5 file. We provide the mergeH5 function for this purpose (see below for details)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from anndata import read_h5ad\n",
    "ctrl_dge = read_h5ad('./src/pyliger/datasets/pbmc_ctrl.h5ad', backed='r+')\n",
    "stim_dge = read_h5ad('./src/pyliger/datasets/pbmc_stim.h5ad', backed='r+')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adata_list = [stim_dge, ctrl_dge]\n",
    "pbmcs = pyliger.create_liger(adata_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then perform the normalization, gene selection, and gene scaling in an online fashion, reading the data from disk in small batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pyliger.normalize(pbmcs)\n",
    "pyliger.select_genes(pbmcs, var_thresh=0.2, do_plot=False)\n",
    "pyliger.scale_not_center(pbmcs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "W = np.loadtxt('/Users/lulu/Desktop/W1.txt')\n",
    "V1 = np.loadtxt('/Users/lulu/Desktop/V1.txt')\n",
    "V2 = np.loadtxt('/Users/lulu/Desktop/V2.tx t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "W.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pbmcs.adata_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Online Integrative Nonnegative Matrix Factorization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use online iNMF to factorize the data, again using only minibatches that we read from the HDF5 files on demand (default mini-batch size = 5000). Sufficient number of iterations is crucial for obtaining ideal factorization result. If the size of the mini-batch is set to be close to the size of the whole dataset (i.e. an epoch only contains one iteration), max.epochs needs to be increased accordingly for more iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pyliger.online_iNMF(pbmcs, k = 20, miniBatch_size = 5000, max_epochs = 5, W_init=W, V_init=[V1, V2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#%%timeit -n1 -r10\n",
    "pyliger.online_iNMF(pbmcs, k = 20, miniBatch_size = 5000, max_epochs = 5, verbose=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quantile Normalization and Downstream Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After performing the factorization, we can perform quantile normalization to align the datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pyliger.quantile_norm(pbmcs, max_sample=2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also visualize the cell factor loadings in two dimensions using t-SNE or UMAP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pyliger.run_umap(pbmcs, distance = 'cosine', n_neighbors = 30, min_dist = 0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "all_plots = pyliger.plot_by_dataset_and_cluster(pbmcs, axis_labels = ['UMAP 1', 'UMAP 2'], return_plots = True)\n",
    "all_plots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scenario 2: iterative refinement by incorporating new datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Please load following modules\n",
    "import pyliger\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import scipy.io\n",
    "from scipy.sparse import csr_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load Dataset\n",
    "allen_smarter_cells = pyliger.datasets.allen_smarter_cells(backed='r+')\n",
    "allen_smarter_nuclei = pyliger.datasets.allen_smarter_nuclei(backed='r+')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "MOp = pyliger.create_liger([allen_smarter_cells])\n",
    "pyliger.normalize(MOp)\n",
    "pyliger.select_genes(MOp, var_thresh=2)\n",
    "pyliger.scale_not_center(MOp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pyliger.online_iNMF(MOp, k = 40, max_epochs = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pyliger.quantile_norm(MOp)\n",
    "pyliger.run_umap(MOp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "all_plots = pyliger.plot_by_dataset_and_cluster(MOp, axis_labels = ['UMAP 1', 'UMAP 2'], return_plots = True)\n",
    "all_plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "pyliger.online_iNMF(MOp, X_new = [allen_smarter_nuclei], k = 40, max_epochs=1, V_init=[V2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pyliger.quantile_norm(MOp, max_sample=2000)\n",
    "pyliger.run_umap(MOp, use_raw=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "all_plots = pyliger.plot_by_dataset_and_cluster(MOp, axis_labels = ['UMAP 1', 'UMAP 2'], return_plots = True)\n",
    "all_plots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scenario 3: projecting new datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Please load following modules\n",
    "import pyliger\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import scipy.io\n",
    "from scipy.sparse import csr_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load Dataset\n",
    "allen_smarter_cells = pyliger.datasets.allen_smarter_cells(backed='r+')\n",
    "allen_smarter_nuclei = pyliger.datasets.allen_smarter_nuclei(backed='r+')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MOp = pyliger.create_liger([allen_smarter_cells])\n",
    "pyliger.normalize(MOp)\n",
    "pyliger.select_genes(MOp, var_thresh=2)\n",
    "pyliger.scale_not_center(MOp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pyliger.online_iNMF(MOp, k = 40, max_epochs = 1, W_init=W, V_init=[V1])\n",
    "pyliger.quantile_norm(MOp, max_sample=2000)\n",
    "pyliger.run_umap(MOp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "all_plots = pyliger.plot_by_dataset_and_cluster(MOp, axis_labels = ['UMAP 1', 'UMAP 2'], return_plots = True)\n",
    "all_plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_plots[1].save(filename = '/Users/lulu/Desktop/1.png', height=12, width=10, units = 'in', dpi=500)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}