Updated function 'read_10X_h5'

Updated function 'read_10X_h5' and added info into the tutorial.

integrating_multi_scRNA_data.ipynb

@@ -16,7 +16,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 2,
    "metadata": {
     "scrolled": true
    },
@@ -37,32 +37,49 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "1. For the first portion of this protocol, we will be integrating data from control and interferon-stimulated PBMCs from Kang et al, 2017. The data can be found in the Gene Expression Omnibus, Series GSE96583. This dataset was originally in the form of output from the 10X Cellranger pipeline, though we will directly load downsampled versions of the control and stimulated DGEs here.\n",
+    "**1.** For the first portion of this protocol, we will be integrating data from control and interferon-stimulated PBMCs from Kang et al, 2017. The data can be found in the Gene Expression Omnibus, Series GSE96583. This dataset was originally in the form of output from the 10X Cellranger pipeline, though we will directly load downsampled versions of the control and stimulated DGEs here.\n",
     "\n",
-    "    For convenience, we have prepared the pre-processed data which are ready to use. There are three datasets: “PBMC_control.h5ad” and “PBMC_interferon-stimulated.h5ad”, which correspond to control and interferon-stimulated PBMCs individually. "
+    "**For convenience, we have prepared the pre-processed data which are ready to use.** There are three datasets: “PBMC_control.h5ad” and “PBMC_interferon-stimulated.h5ad”, which correspond to control and interferon-stimulated PBMCs individually. The data can be downloaded [here](https://www.dropbox.com/sh/vybwlxe1wmcvewv/AAADGoqSWDKgKc5JaoVLxxg7a?dl=0)."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 3,
    "metadata": {},
    "outputs": [],
    "source": [
     "from anndata import read_h5ad\n",
-    "ctrl_dge = read_h5ad('./src/pyliger/datasets/pbmc_ctrl_downsampled.h5ad')\n",
-    "stim_dge = read_h5ad('./src/pyliger/datasets/pbmc_stim_downsampled.h5ad')"
+    "ctrl_dge = read_h5ad('WHERE_STORES/pbmc_ctrl_downsampled.h5ad')\n",
+    "stim_dge = read_h5ad('WHERE_STORES/pbmc_stim_downsampled.h5ad')"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "2. With the digital gene expression matrices for both datasets, we can initialize a pyliger object using create_liger function."
+    "For 10X CellRanger output (.h5), we can instead use the `read_10X_h5` function, which generates an AnnData object directly from the output directory."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ctrl_dge = pyliger.read_10X_h5(sample_dir='DIRECTORY_WHERE_STORES_CTRL_h5', sample_name='ctrl_dge')\n",
+    "stim_dge = pyliger.read_10X_h5(sample_dir='DIRECTORY_WHERE_STORES_STIM_h5', sample_name='stim_dge')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**2.** With the digital gene expression matrices for both datasets, we can initialize a pyliger object using `create_liger` function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
    "metadata": {
     "scrolled": true
    },
@@ -85,19 +102,19 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "ifnb_liger now contains two datasets in its adata_list attribute. Each dataset is an AnnData object. We can run the rest of the analysis on this pyliger object."
+    "`ifnb_liger` now contains two datasets in its adata_list attribute. Each dataset is an AnnData object. We can run the rest of the analysis on this pyliger object."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "3. Before we can run iNMF on our datasets, we must run several preprocessing steps to normalize expression data to account for differences in sequencing depth and efficiency between cells, identify variably expressed genes, and scale the data so that each gene has the same variance. Note that because nonnegative matrix factorization requires positive values, we do not center the data by subtracting the mean. We also do not log transform the data."
+    "**3.** Before we can run iNMF on our datasets, we must run several preprocessing steps to normalize expression data to account for differences in sequencing depth and efficiency between cells, identify variably expressed genes, and scale the data so that each gene has the same variance. Note that because nonnegative matrix factorization requires positive values, we do not center the data by subtracting the mean. We also do not log transform the data."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 5,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -117,19 +134,19 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "4. We are now able to run integrative non-negative matrix factorization on the normalized and scaled datasets. The key parameter for this analysis is k, the number of matrix factors (analogous to the number of principal components in PCA). In general, we find that a value of k between 20 and 40 is suitable for most analyses and that results are robust for choice of k. Because LIGER is an unsupervised, exploratory approach, there is no single “right” value for k, and in practice, users choose k from a combination of biological prior knowledge and other information."
+    "**4.** We are now able to run integrative non-negative matrix factorization on the normalized and scaled datasets. The key parameter for this analysis is k, the number of matrix factors (analogous to the number of principal components in PCA). In general, we find that a value of k between 20 and 40 is suitable for most analyses and that results are robust for choice of k. Because LIGER is an unsupervised, exploratory approach, there is no single “right” value for k, and in practice, users choose k from a combination of biological prior knowledge and other information."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 6,
    "metadata": {},
    "outputs": [
     {
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "100%|██████████| 30/30 [00:51<00:00,  1.70s/it]\n"
+      "100%|██████████| 30/30 [00:50<00:00,  1.67s/it]\n"
      ]
     }
    ],
@@ -143,8 +160,8 @@
    "source": [
     "Important parameters are as follows:\n",
     "\n",
-    "    -k: Integer value specifying the inner dimension of factorization, or number of factors. Higher k is recommended for datasets with more substructure. We find that a value of k in the range 20 - 40 works well for most datasets. Because this is an unsupervised, exploratory analysis, there is no single “right” value for k, and in practice, users choose k from a combination of biological prior knowledge and other information.\n",
-    "    -lambda: This is a regularization parameter. Larger values penalize dataset-specific effects more strongly, causing the datasets to be better aligned, but possibly at the cost of higher reconstruction error. The default value is 5. We recommend using this value for most analyses, but find that it can be lowered to 1 in cases where the dataset differences are expected to be relatively small, such as scRNA-seq data from the same tissue but different individuals.\n",
+    "- k: Integer value specifying the inner dimension of factorization, or number of factors. Higher k is recommended for datasets with more substructure. We find that a value of k in the range 20 - 40 works well for most datasets. Because this is an unsupervised, exploratory analysis, there is no single “right” value for k, and in practice, users choose k from a combination of biological prior knowledge and other information.\n",
+    "- lambda: This is a regularization parameter. Larger values penalize dataset-specific effects more strongly, causing the datasets to be better aligned, but possibly at the cost of higher reconstruction error. The default value is 5. We recommend using this value for most analyses, but find that it can be lowered to 1 in cases where the dataset differences are expected to be relatively small, such as scRNA-seq data from the same tissue but different individuals.\n",
     "    -thresh: This sets the convergence threshold. Lower values cause the algorithm to run longer. The default is 1e-6.\n",
     "    -max_iters: This variable sets the maximum number of iterations to perform. The default value is 30.\n",
     "    \n",
@@ -164,12 +181,12 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "5. We can now use the resulting factors to jointly cluster cells and perform quantile normalization by dataset, factor, and cluster to fully integrate the datasets. All of this functionality is encapsulated within the quantile_norm function, which uses max factor assignment followed by refinement using a k-nearest neighbors graph."
+    "**5.** We can now use the resulting factors to jointly cluster cells and perform quantile normalization by dataset, factor, and cluster to fully integrate the datasets. All of this functionality is encapsulated within the quantile_norm function, which uses max factor assignment followed by refinement using a k-nearest neighbors graph."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 7,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -181,28 +198,27 @@
    "metadata": {},
    "source": [
     "Important parameters of quantile_norm are as follows:\n",
-    "\n",
-    "    -knn_k This sets the number of nearest neighbors for within-dataset KNN graph. The default is 20.\n",
-    "    -quantiles This sets the number of quantiles to use for quantile normalization. The default is 50.\n",
-    "    -min_cells This indicates the minimum number of cells to consider a cluster as shared across datasets. The default is 20.\n",
-    "    -dims_use This sets the indices of factors to use for quantile normalization. The user can pass in a vector of indices indicating specific factors. This is helpful for excluding factors capturing biological signals such as the cell cycle or technical signals such as mitochondrial genes. The default is all k of the factors.\n",
-    "    -do_center This indicates whether to center the data when scaling factors. The default is FALSE. This option should be set to TRUE when metagene loadings have a mean above zero, as with dense data such as DNA methylation.\n",
-    "    -max_sample This sets the maximum number of cells used for quantile normalization of each cluster and factor. The default is 1000.\n",
-    "    -refine_knn This indicates whether to increase robustness of cluster assignments using KNN graph. The default is TRUE.\n",
-    "    -eps This sets the error bound of the nearest neighbor search. The default is 0.9. Lower values give more accurate nearest neighbor graphs but take much longer to computer.\n",
-    "    -ref_dataset This indicates the name of the dataset to be used as a reference for quantile normalization. By default, the dataset with the largest number of cells is used."
+    "- knn_k This sets the number of nearest neighbors for within-dataset KNN graph. The default is 20.\n",
+    "- quantiles This sets the number of quantiles to use for quantile normalization. The default is 50.\n",
+    "- min_cells This indicates the minimum number of cells to consider a cluster as shared across datasets. The default is 20.\n",
+    "- dims_use This sets the indices of factors to use for quantile normalization. The user can pass in a vector of indices indicating specific factors. This is helpful for excluding factors capturing biological signals such as the cell cycle or technical signals such as mitochondrial genes. The default is all k of the factors.\n",
+    "- do_center This indicates whether to center the data when scaling factors. The default is FALSE. This option should be set to TRUE when metagene loadings have a mean above zero, as with dense data such as DNA methylation.\n",
+    "- max_sample This sets the maximum number of cells used for quantile normalization of each cluster and factor. The default is 1000.\n",
+    "- refine_knn This indicates whether to increase robustness of cluster assignments using KNN graph. The default is TRUE.\n",
+    "- eps This sets the error bound of the nearest neighbor search. The default is 0.9. Lower values give more accurate nearest neighbor graphs but take much longer to computer.\n",
+    "- ref_dataset This indicates the name of the dataset to be used as a reference for quantile normalization. By default, the dataset with the largest number of cells is used."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "6. The quantile_norm procedure produces joint clustering assignments and a low-dimensional representation that integrates the datasets together. These joint clusters directly from iNMF can be used for downstream analyses (see below). Alternatively, you can also run Louvain community detection, an algorithm commonly used for single-cell data, on the normalized cell factors. The Louvain algorithm excels at merging small clusters into broad cell classes and thus may be more desirable in some cases than the maximum factor assignments produced directly by iNMF."
+    "**6.** The `quantile_norm` procedure produces joint clustering assignments and a low-dimensional representation that integrates the datasets together. These joint clusters directly from iNMF can be used for downstream analyses (see below). Alternatively, you can also run Louvain community detection, an algorithm commonly used for single-cell data, on the normalized cell factors. The Louvain/Leiden algorithm excels at merging small clusters into broad cell classes and thus may be more desirable in some cases than the maximum factor assignments produced directly by iNMF."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 8,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -221,12 +237,13 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "7. To visualize the clustering of cells graphically, we can project the normalized cell factors to two or three dimensions. Liger supports both t-SNE and UMAP for this purpose. Note that if both techniques are run, the object will only hold the results from the most recent."
+    "## Visualization\n",
+    "**7.** To visualize the clustering of cells graphically, we can project the normalized cell factors to two or three dimensions. Liger supports both t-SNE and UMAP for this purpose. Note that if both techniques are run, the object will only hold the results from the most recent."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 9,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -237,12 +254,19 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "8. plot_by_dataset_and_cluster returns two graphs, generated by t-SNE or UMAP in the previous step. The first colors cells by dataset of origin, and the second by cluster as determined by Liger. The plots provide visual confirmation that the datasets are well aligned and the clusters are consistent with the shape of the data as revealed by UMAP."
+    "The `PyLiger` package implements a variety of utilities for visualization and analysis of clustering, gene expression across datasets, and comparisons of cluster assignments. We will summarize several here."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**8.** `plot_by_dataset_and_cluster` returns two graphs, generated by t-SNE or UMAP in the previous step. The first colors cells by dataset of origin, and the second by cluster as determined by Liger. The plots provide visual confirmation that the datasets are well aligned and the clusters are consistent with the shape of the data as revealed by UMAP."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 10,
    "metadata": {},
    "outputs": [
     {
@@ -2238,10 +2262,10 @@
     {
      "data": {
       "text/plain": [
-       "[<ggplot: (8779610716076)>, <ggplot: (8779610598000)>]"
+       "[<ggplot: (8781923116403)>, <ggplot: (8781924104349)>]"
       ]
      },
-     "execution_count": 12,
+     "execution_count": 10,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -4269,7 +4293,8 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Using the runWilcoxon function, we can next identify gene markers for all clusters. We can also compare expression within each cluster across datasets, which in this case reveals markers of interferon-beta stimulation. The function returns a table of data that allows us to determine the significance of each gene’s differential expression, including log fold change, area under the curve and p-value."
+    "## Downstream Analysis\n",
+    "**9.** Using the `run_wilcoxon` function, we can next identify gene markers for all clusters. We can also compare expression within each cluster across datasets, which in this case reveals markers of interferon-beta stimulation. The function returns a table of data that allows us to determine the significance of each gene’s differential expression, including log fold change, area under the curve and p-value."
    ]
   },
   {
@@ -4483,7 +4508,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# GO"
+    "**10.** GO"
    ]
   },
   {

pyliger/_utilities.py

@@ -1,3 +1,6 @@
+import os
+import logging
+import h5sparse
 import numpy as np
 
 from anndata import AnnData
@@ -71,6 +74,7 @@ def _remove_missing_obs(adata,
     else:
         missing = np.ravel(np.sum(filter_data, axis=0)) == 0
     if np.sum(missing) > 0:
+        #logging.info('Removing {} {} not expressing{} in {}.'.format(np.sum(missing), removed, expressed, data_type))
         print('Removing {} {} not expressing{} in {}.'.format(np.sum(missing), removed, expressed, data_type))
         if use_rows:
             # show gene name when the total of missing is less than 25
@@ -90,7 +94,10 @@ def _remove_missing_obs(adata,
 def _h5_idx_generator(chunk_size, matrix_size):
     """ """
     previous_idx = 0
-    current_idx = chunk_size
+    if matrix_size < chunk_size:
+        current_idx = matrix_size
+    else:
+        current_idx = chunk_size
     num_chunk = np.ceil(matrix_size/chunk_size).astype(int)
     iter = 0
     while current_idx <= matrix_size and iter < num_chunk:
@@ -116,3 +123,18 @@ def nonneg(x, eps=1e-16):
     """ Given a input matrix, set all negative values to be zero """
     x[x < eps] = eps
     return x
+
+
+def _create_h5_using_adata(adata, chunk_size):
+    # create h5 file.
+    if not os.path.isdir('./results'):
+        os.mkdir('./results')
+
+    file_name = './results/' + adata.uns['sample_name'] + '.hdf5'
+    with h5sparse.File(file_name, 'w') as f:
+        for left, right in _h5_idx_generator(chunk_size, adata.shape[0]):
+            if 'raw_data' not in f.keys():
+                f.create_dataset('raw_data', data=adata[left:right, :].X, chunks=(chunk_size,), maxshape=(None,))
+            else:
+                f['raw_data'].append(adata[left:right, :].X)
+    return None