diff --git a/CHANGELOG.md b/CHANGELOG.md index b64db49f..a35406da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -101,6 +101,7 @@ few days ago, so I doubt this has been a big problem for anyone. ### Performance enhancements ### Miscellaneous +- Added a draft "Moving Pictures" tutorial. - When replacing certain characters within column names, Qurro no longer replaces `'` or `"` characters with \| characters. Instead, `'` and `"` characters are just removed from column names entirely. @@ -110,14 +111,14 @@ - Replaced uses of `log ratio` with `log-ratio` throughout Qurro's code and documentation. - On the [Qurro website](https://biocore.github.io/qurro/): - - Added a "Mackerel" demo (based on the output of [this](https://github.com/knightlab-analyses/qurro-mackerel-analysis/) - analysis). + - Added a "Mackerel" demo (based on the output of [this](https://github.com/knightlab-analyses/qurro-mackerel-analysis/) analysis), and associated with data from [this paper](https://www.biorxiv.org/content/10.1101/721555v1). - Replaced "OTUs" for the Moving Pictures and Sleep Apnea demo dataset feature types with "ASVs / sOTUs". - Replaced "OTUs" for the Byrd dataset feature type with "Taxa". - Cleaned up the Byrd demo's dataset citation (to reflect the fact that the Morton/Marotz et al. 2019 paper is now published). -- Added a draft "Moving Pictures" tutorial. + - Removed the now-incomplete "tutorials" section, in favor of just linking to + Qurro's GitHub page. - DEICODE ordinations used in demos and test inputs have been rerun with DEICODE version 0.2.3. ([#188](https://github.com/biocore/qurro/issues/188)) diff --git a/README.md b/README.md index 7caf65fc..b9c36663 100644 --- a/README.md +++ b/README.md @@ -90,8 +90,8 @@ available in the QIIME 2 plugin version of Qurro. In the style of the QIIME 2 and DEICODE moving pictures tutorials, there is a draft moving pictures tutorial (showing how to use Qurro within QIIME 2, and -how to interact with the generated Qurro visualization) available -[here](https://github.com/biocore/qurro/blob/master/docs/tutorials/moving-pictures/moving-pictures.md). +how to interact with the generated Qurro visualization) **available +[here](https://github.com/biocore/qurro/blob/master/docs/tutorials/moving-pictures/moving-pictures.md).** This tutorial is still a work in progress, so feel free to contact us if you have any outstanding questions (or any suggestions for improving this tutorial). @@ -160,6 +160,9 @@ Qurro also uses [Travis-CI](https://travis-ci.org/) and ### Data Sources +The test data located in `qurro/tests/input/mackerel/` were exported from +QIIME 2 artifacts in [this repository](https://github.com/knightlab-analyses/qurro-mackerel-analysis). These data are from Minich et al. 2019 [1]. + The test data located in `qurro/tests/input/byrd/` are from [this repository](https://github.com/knightlab-analyses/reference-frames). These data, in turn, originate from Byrd et al.'s 2017 study on atopic @@ -203,7 +206,10 @@ And thanks to a bunch of the Knight Lab for helping name the tool :) ## References -[1] Becker, R. A. & Cleveland, W. S. (1987). Brushing scatterplots. _Technometrics, 29_(2), 127-142. (Section 4.1 in particular talks about linking visualizations.) +[1] Minich, J. J., Petrus, S., Michael, J. D., Michael, T. P., Knight, R., & +Allen, E. E. (2019). Temporal, environmental, and biological +drivers of the mucosal microbiome in a wild marine fish, Scomber japonicus. +_Manuscript under review._ [2] Byrd, A. L., Deming, C., Cassidy, S. K., Harrison, O. J., Ng, W. I., Conlan, S., ... & NISC Comparative Sequencing Program. (2017). Staphylococcus aureus and Staphylococcus epidermidis strain diversity underlying pediatric atopic dermatitis. _Science translational medicine, 9_(397), eaal4651. diff --git a/docs/index.html b/docs/index.html index c413eaa2..fd9da91b 100644 --- a/docs/index.html +++ b/docs/index.html @@ -14,15 +14,17 @@

Qurro: Quantitative Rank/Ratio Observations

+

A software tool created by the Knight Lab at UC San Diego. - - For information about what Qurro does, see its + >. For details, please see Qurro's GitHub page.

+

+ This page contains demos of Qurro on various datasets. +

Demos

@@ -102,10 +104,8 @@

Demos

Songbird Here - Minich et al. 2019 (under review); - Minich et al. 2018Minich et al. 2019 (under review); differentials generated Demos
- -

Tutorials

-

- See - here - for example Jupyter Notebooks that show how to generate Qurro plots - from inside and outside of QIIME 2. -

diff --git a/docs/tutorials/moving-pictures/data/qurro-plot.qzv b/docs/tutorials/moving-pictures/data/qurro-plot.qzv index 3acaa0a6..913e970f 100644 Binary files a/docs/tutorials/moving-pictures/data/qurro-plot.qzv and b/docs/tutorials/moving-pictures/data/qurro-plot.qzv differ diff --git a/docs/tutorials/moving-pictures/moving-pictures.md b/docs/tutorials/moving-pictures/moving-pictures.md index c0db7215..a373fefe 100644 --- a/docs/tutorials/moving-pictures/moving-pictures.md +++ b/docs/tutorials/moving-pictures/moving-pictures.md @@ -1,4 +1,5 @@ # Qurro QIIME 2 "Moving Pictures" Tutorial +Using Qurro v0.3.0. ## Introduction @@ -22,8 +23,8 @@ compared to sample metadata categories. Both analyzing the log-ratios between features' abundances; Qurro provides an easy way to do this. -Log-ratio analyses are needed because data obtained from a microbiome study is -inherently compositional: we only have access to the relative abundances of +Log-ratio analyses are needed because data obtained from a microbiome study is, +in general, "compositional": we only have access to the relative abundances of features in each sample, instead of their absolute abundances. To quote [Gloor et al. 2017](https://www.frontiersin.org/articles/10.3389/fmicb.2017.02224): @@ -82,8 +83,8 @@ necessary files for this tutorial here: - `table.qza` [view](https://view.qiime2.org/?src=https%3A%2F%2Fdocs.qiime2.org%2F2019.4%2Fdata%2Ftutorials%2Fmoving-pictures%2Ftable.qza) | [download](https://docs.qiime2.org/2019.4/data/tutorials/moving-pictures/table.qza) - `sample-metadata.tsv` [download](https://data.qiime2.org/2019.4/tutorials/moving-pictures/sample_metadata.tsv) - `taxonomy.qza` [view](https://view.qiime2.org/?src=https%3A%2F%2Fdocs.qiime2.org%2F2019.4%2Fdata%2Ftutorials%2Fmoving-pictures%2Ftaxonomy.qza) | [download](https://docs.qiime2.org/2019.4/data/tutorials/moving-pictures/taxonomy.qza) -- `ordination.qza` [view](https://view.qiime2.org/?src=http%3A%2F%2Fbiocore.github.io%2Fqurro%2Ftutorials%2Fmoving-pictures%2Fdata%2Fordination.qza) | [download](http://biocore.github.io/qurro/tutorials/moving-pictures/data/ordination.qza) -- `biplot.qzv` [view](https://view.qiime2.org/?src=http%3A%2F%2Fbiocore.github.io%2Fqurro%2Ftutorials%2Fmoving-pictures%2Fdata%2Fbiplot.qzv) | [download](http://biocore.github.io/qurro/tutorials/moving-pictures/data/biplot.qzv) +- `ordination.qza` [view](https://view.qiime2.org/?src=https%3A%2F%2Fbiocore.github.io%2Fqurro%2Ftutorials%2Fmoving-pictures%2Fdata%2Fordination.qza) | [download](https://biocore.github.io/qurro/tutorials/moving-pictures/data/ordination.qza) +- `biplot.qzv` [view](https://view.qiime2.org/?src=https%3A%2F%2Fbiocore.github.io%2Fqurro%2Ftutorials%2Fmoving-pictures%2Fdata%2Fbiplot.qzv) | [download](https://biocore.github.io/qurro/tutorials/moving-pictures/data/biplot.qzv) ## Running Qurro @@ -102,7 +103,7 @@ qiime qurro loading-plot \ ##### Output Artifacts -- `qurro-plot.qzv` [view](https://view.qiime2.org/?src=http%3A%2F%2Fbiocore.github.io%2Fqurro%2Ftutorials%2Fmoving-pictures%2Fdata%2Fqurro-plot.qzv) | [download](http://biocore.github.io/qurro/tutorials/moving-pictures/data/qurro-plot.qzv) +- `qurro-plot.qzv` [view](https://view.qiime2.org/?src=https%3A%2F%2Fbiocore.github.io%2Fqurro%2Ftutorials%2Fmoving-pictures%2Fdata%2Fqurro-plot.qzv) | [download](https://biocore.github.io/qurro/tutorials/moving-pictures/data/qurro-plot.qzv) You just generated your first Qurro plot! `qurro-plot.qzv` is a `.qzv` file -- in other words, a QIIME 2 visualization. You can view it either by running @@ -265,7 +266,8 @@ you can do this using `qiime tools view` or by uploading `biplot.qzv` to [view.qiime2.org](https://view.qiime2.org/). Here's a screenshot of the biplot, with samples colored by their `BodySite` and -features (arrows) colored by their `Taxon`. +features (arrows) colored by their `Taxon`. (Feature labels have been hidden in +this screenshot.) Screenshot of the biplot generated in the DEICODE tutorial, visualized in Emperor. diff --git a/docs/tutorials/moving-pictures/screenshots/emperor1.png b/docs/tutorials/moving-pictures/screenshots/emperor1.png index 2ecc8b89..1e8745af 100644 Binary files a/docs/tutorials/moving-pictures/screenshots/emperor1.png and b/docs/tutorials/moving-pictures/screenshots/emperor1.png differ diff --git a/docs/tutorials/moving-pictures/screenshots/qurro1.png b/docs/tutorials/moving-pictures/screenshots/qurro1.png index 44a9b7ee..3f2692a1 100644 Binary files a/docs/tutorials/moving-pictures/screenshots/qurro1.png and b/docs/tutorials/moving-pictures/screenshots/qurro1.png differ diff --git a/docs/tutorials/moving-pictures/screenshots/qurro2.png b/docs/tutorials/moving-pictures/screenshots/qurro2.png index 10d4322a..58104ab3 100644 Binary files a/docs/tutorials/moving-pictures/screenshots/qurro2.png and b/docs/tutorials/moving-pictures/screenshots/qurro2.png differ diff --git a/docs/tutorials/moving-pictures/screenshots/qurro3.png b/docs/tutorials/moving-pictures/screenshots/qurro3.png index 954e018e..d7c40f4c 100644 Binary files a/docs/tutorials/moving-pictures/screenshots/qurro3.png and b/docs/tutorials/moving-pictures/screenshots/qurro3.png differ diff --git a/example_notebooks/DEICODE_sleep_apnea/deicode_example.ipynb b/example_notebooks/DEICODE_sleep_apnea/deicode_example.ipynb index b0f465c8..31e9c016 100644 --- a/example_notebooks/DEICODE_sleep_apnea/deicode_example.ipynb +++ b/example_notebooks/DEICODE_sleep_apnea/deicode_example.ipynb @@ -157,7 +157,7 @@ "Usage: \u001b[34mqiime qurro loading-plot\u001b[0m [OPTIONS]\r\n", "\r\n", " Generates an interactive visualization of feature loadings in tandem with\r\n", - " a visualization of the log ratios of selected features' sample abundances.\r\n", + " a visualization of the log-ratios of selected features' sample abundances.\r\n", "\r\n", "\u001b[1mInputs\u001b[0m:\r\n", " \u001b[34m\u001b[4m--i-ranks\u001b[0m ARTIFACT \u001b[32mPCoAResults % Properties('biplot')\u001b[0m\r\n", @@ -178,17 +178,13 @@ " be merged) \u001b[35m[optional]\u001b[0m\r\n", " \u001b[34m--p-extreme-feature-count\u001b[0m INTEGER\r\n", " If specified, Qurro will only use this many \"extreme\"\r\n", - " features from either end of all of the rankings. This\r\n", - " is useful when dealing with huge datasets (e.g. with\r\n", - " BIOM tables exceeding 1 million entries), for which\r\n", - " running Qurro normally might take a long amount of time\r\n", - " or crash due to memory limits. This value must be at\r\n", - " least 1, and must be an integer. Note that samples\r\n", - " without any observed features after this filtering step\r\n", - " will be removed from the Qurro visualization, as will\r\n", - " features without any observations in samples (even\r\n", - " features that were not filtered due to being\r\n", - " sufficiently \"extreme\"). \u001b[35m[optional]\u001b[0m\r\n", + " features from both ends of all of the rankings. This is\r\n", + " useful when dealing with huge datasets (e.g. with BIOM\r\n", + " tables exceeding 1 million entries), for which running\r\n", + " Qurro normally might take a long amount of time or\r\n", + " crash due to memory limits. Note that the automatic\r\n", + " removal of empty samples and features from the table\r\n", + " will be done *after* this filtering step. \u001b[35m[optional]\u001b[0m\r\n", " \u001b[34m--p-debug\u001b[0m / \u001b[34m--p-no-debug\u001b[0m\r\n", " If this flag is used, Qurro will output debug\r\n", " messages. Note that you'll also need to use the\r\n", @@ -317,20 +313,15 @@ " necessary. [required]\r\n", " -x, --extreme-feature-count INTEGER\r\n", " If specified, Qurro will only use this many\r\n", - " \"extreme\" features from either end of all of\r\n", + " \"extreme\" features from both ends of all of\r\n", " the rankings. This is useful when dealing\r\n", " with huge datasets (e.g. with BIOM tables\r\n", " exceeding 1 million entries), for which\r\n", " running Qurro normally might take a long\r\n", " amount of time or crash due to memory\r\n", - " limits. This value must be at least 1, and\r\n", - " must be an integer. Note that samples\r\n", - " without any observed features after this\r\n", - " filtering step will be removed from the\r\n", - " Qurro visualization, as will features\r\n", - " without any observations in samples (even\r\n", - " features that were not filtered due to being\r\n", - " sufficiently \"extreme\").\r\n", + " limits. Note that the automatic removal of\r\n", + " empty samples and features from the table\r\n", + " will be done *after* this filtering step.\r\n", " -gnps, --assume-gnps-feature-metadata\r\n", " If specified, Qurro will assume that the\r\n", " input feature metadata was obtained from\r\n", diff --git a/example_notebooks/songbird_red_sea/songbird_example.ipynb b/example_notebooks/songbird_red_sea/songbird_example.ipynb index e8f31ba3..9c606e88 100644 --- a/example_notebooks/songbird_red_sea/songbird_example.ipynb +++ b/example_notebooks/songbird_red_sea/songbird_example.ipynb @@ -25,7 +25,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 9, "metadata": {}, "outputs": [], "source": [ @@ -51,7 +51,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 10, "metadata": {}, "outputs": [ { @@ -83,7 +83,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 11, "metadata": {}, "outputs": [ { @@ -113,12 +113,12 @@ "metadata": {}, "source": [ "### 1. B. Using Qurro through QIIME 2\n", - "Since we're working with Songbird output, we use the `qiime qurro differential-plot` command." + "Since we're working with Songbird output (feature differentials), we use the `qiime qurro differential-plot` command." ] }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 12, "metadata": {}, "outputs": [ { @@ -150,17 +150,13 @@ " be merged) \u001b[35m[optional]\u001b[0m\r\n", " \u001b[34m--p-extreme-feature-count\u001b[0m INTEGER\r\n", " If specified, Qurro will only use this many \"extreme\"\r\n", - " features from either end of all of the rankings. This\r\n", - " is useful when dealing with huge datasets (e.g. with\r\n", - " BIOM tables exceeding 1 million entries), for which\r\n", - " running Qurro normally might take a long amount of time\r\n", - " or crash due to memory limits. This value must be at\r\n", - " least 1, and must be an integer. Note that samples\r\n", - " without any observed features after this filtering step\r\n", - " will be removed from the Qurro visualization, as will\r\n", - " features without any observations in samples (even\r\n", - " features that were not filtered due to being\r\n", - " sufficiently \"extreme\"). \u001b[35m[optional]\u001b[0m\r\n", + " features from both ends of all of the rankings. This is\r\n", + " useful when dealing with huge datasets (e.g. with BIOM\r\n", + " tables exceeding 1 million entries), for which running\r\n", + " Qurro normally might take a long amount of time or\r\n", + " crash due to memory limits. Note that the automatic\r\n", + " removal of empty samples and features from the table\r\n", + " will be done *after* this filtering step. \u001b[35m[optional]\u001b[0m\r\n", " \u001b[34m--p-debug\u001b[0m / \u001b[34m--p-no-debug\u001b[0m\r\n", " If this flag is used, Qurro will output debug\r\n", " messages. Note that you'll also need to use the\r\n", @@ -185,7 +181,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 13, "metadata": {}, "outputs": [ { @@ -227,34 +223,34 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/songbird/util.py:122: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\\t'.\n", + "/anaconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/util.py:122: FutureWarning: read_table is deprecated, use read_csv instead, passing sep='\\t'.\n", " filepath, dtype=object)\n", - "WARNING:tensorflow:From /anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/songbird/multinomial.py:75: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.\n", + "WARNING:tensorflow:From /anaconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/multinomial.py:75: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.random.categorical instead.\n", - "WARNING:tensorflow:From /anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n", + "WARNING:tensorflow:From /anaconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Colocations handled automatically by placer.\n", - "WARNING:tensorflow:From /anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/songbird/multinomial.py:91: Normal.__init__ (from tensorflow.python.ops.distributions.normal) is deprecated and will be removed after 2019-01-01.\n", + "WARNING:tensorflow:From /anaconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/multinomial.py:91: Normal.__init__ (from tensorflow.python.ops.distributions.normal) is deprecated and will be removed after 2019-01-01.\n", "Instructions for updating:\n", "The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.\n", - "WARNING:tensorflow:From /anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/tensorflow/python/ops/distributions/normal.py:160: Distribution.__init__ (from tensorflow.python.ops.distributions.distribution) is deprecated and will be removed after 2019-01-01.\n", + "WARNING:tensorflow:From /anaconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/tensorflow/python/ops/distributions/normal.py:160: Distribution.__init__ (from tensorflow.python.ops.distributions.distribution) is deprecated and will be removed after 2019-01-01.\n", "Instructions for updating:\n", "The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.\n", - "WARNING:tensorflow:From /anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/songbird/multinomial.py:100: Multinomial.__init__ (from tensorflow.python.ops.distributions.multinomial) is deprecated and will be removed after 2019-01-01.\n", + "WARNING:tensorflow:From /anaconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/songbird/multinomial.py:100: Multinomial.__init__ (from tensorflow.python.ops.distributions.multinomial) is deprecated and will be removed after 2019-01-01.\n", "Instructions for updating:\n", "The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.\n", - "WARNING:tensorflow:From /anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n", + "WARNING:tensorflow:From /anaconda3/envs/qiime2-2019.7/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.cast instead.\n", - "100%|███████████████████████████████████| 80000/80000 [00:50<00:00, 1572.06it/s]\n" + "100%|███████████████████████████████████| 80000/80000 [00:54<00:00, 1478.63it/s]\n" ] } ], @@ -280,7 +276,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 15, "metadata": {}, "outputs": [ { @@ -316,20 +312,15 @@ " necessary. [required]\r\n", " -x, --extreme-feature-count INTEGER\r\n", " If specified, Qurro will only use this many\r\n", - " \"extreme\" features from either end of all of\r\n", + " \"extreme\" features from both ends of all of\r\n", " the rankings. This is useful when dealing\r\n", " with huge datasets (e.g. with BIOM tables\r\n", " exceeding 1 million entries), for which\r\n", " running Qurro normally might take a long\r\n", " amount of time or crash due to memory\r\n", - " limits. This value must be at least 1, and\r\n", - " must be an integer. Note that samples\r\n", - " without any observed features after this\r\n", - " filtering step will be removed from the\r\n", - " Qurro visualization, as will features\r\n", - " without any observations in samples (even\r\n", - " features that were not filtered due to being\r\n", - " sufficiently \"extreme\").\r\n", + " limits. Note that the automatic removal of\r\n", + " empty samples and features from the table\r\n", + " will be done *after* this filtering step.\r\n", " -gnps, --assume-gnps-feature-metadata\r\n", " If specified, Qurro will assume that the\r\n", " input feature metadata was obtained from\r\n", @@ -357,7 +348,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 16, "metadata": {}, "outputs": [ {