FMT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \"Fecal microbiota transplant\" tutorial\n",
    "\n",
    "**_note:_ This guide assumes you have QIIME 2 installed (e.g. using [this procedure](https://docs.qiime2.org/2020.2/install/native/)). To execute the script properly, open this notebook in a Jupyter Notebook from within a conda QIIME 2 environment.**\n",
    "\n",
    "**_note:_ This tutorial is an adaptation of the same [tutorial](https://docs.qiime2.org/2020.2/tutorials/fmt/) that may be found on the official [QIIME 2 docs website](https://docs.qiime2.org/2020.2/). The original tutorial uses the QIIME 2 CLI interface.**\n",
    "\n",
    "Instead of CLI interface, this tutorial uses Artifact API - a Python 3 application programmer interface (API) for QIIME 2. The Artifact API supports interactive computing with QIIME 2 using the Python 3 programming language. The API is automatically generated, and its availability depends on which QIIME 2 plugins are currently installed. It has been optimized for use in the Jupyter Notebook. The Artifact API is a part of the QIIME 2 framework; no additional software needs to be installed to use it.\n",
    "\n",
    "The notebook was tested using the 2020.2 version of QIIME 2.\n",
    "\n",
    "This document is intended to be run after [the moving pictures tutorial](https://docs.qiime2.org/2020.2/tutorials/moving-pictures/). It is designed to introduce a few new ideas, and to be an exercise in applying the tools that were explored in that document.\n",
    "\n",
    "The data used in this tutorial is derived from a [Fecal Microbiome Transplant study](https://microbiomejournal.biomedcentral.com/articles/10.1186/s40168-016-0225-7) where children under the age of 18 with autism and gastrointestinal disorders, as measured by the Autism Diagnostic Interview-Revised (ADI-R) and Gastrointestinal Symptom Rating Scale (GSRS), respectively, were treated with fecal microbiota transplant in attempt to reduce the severity of their behavioral and gastrointestinal symptoms. We tracked changes in their microbiome, several metrics of the severity of autism including the Parent Global Impressions-III (PGI-III) and the Childhood Autism Rating Scale (CARS), and the severity of their gastrointestinal symptoms through their GSRS score over an eighteen week period. The microbiome was tracked through collection of weekly fecal swab samples (collected by swabbing used toilet paper) and less frequent stool samples (collected as whole stool). In the full study, which was a phase 1 clinical trial designed to test safety of the treatment, eighteen individuals received the treatment, and twenty individuals were followed as controls. The controls did not receive the treatment, but were monitored to track normal temporal variation in the gut microbiome. The fecal material that was transplanted during treatment was also sequenced in this study.\n",
    "\n",
    "This tutorial dataset is a subsample of the data generated for this study. It includes data from five individuals who received treatment and five controls. Between six and sixteen samples are included per individual, including stool and fecal swab samples for each individual, and samples before and after FMT treatment. Five samples of the transplanted fecal material are also included.\n",
    "\n",
    "These data were sequenced on two Illumina MiSeq sequencing runs. As in the Moving Pictures tutorial, we’ll use [DADA2](https://www.ncbi.nlm.nih.gov/pubmed/27214047) to perform initial quality control and generate our `FeatureTable[Frequency]` and `FeatureData[Sequence]` objects. However, the DADA2 denoising process is only applicable to a single sequencing run at a time, so we need to run this on a per sequencing run basis and then merge the results. We’ll work through this initial step, and then pose several questions that can be answered as an exercise.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Importing necessary modules\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import qiime2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from qiime2.plugins import demux, dada2, metadata, feature_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating a new directory\n",
    "\n",
    "Create a directory to work in called `qiime2-fmt-tutorial` and change to that directory:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "workdir='/path/to/your/directory/qiime2-fmt-tutorial/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p $workdir\n",
    "!cd $workdir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Obtaining and importing data files\n",
    "\n",
    "As in the Moving Pictures study, you should begin your analysis by familiarizing yourself with the [sample metadata](https://data.qiime2.org/2020.2/tutorials/fmt/sample_metadata.tsv) . You can again access the sample metadata as a Google Spreadsheet. Notice that there are three tabs in this spreadsheet. This first tab (called sample-metadata) contains all of the clinical metadata.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-04-15 09:23:56--  https://data.qiime2.org/2020.2/tutorials/moving-pictures/sample_metadata.tsv\n",
      "Resolving data.qiime2.org (data.qiime2.org)... 52.35.38.247\n",
      "Connecting to data.qiime2.org (data.qiime2.org)|52.35.38.247|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 FOUND\n",
      "Location: https://docs.google.com/spreadsheets/d/15HpBuwlUbm6Yg12qOtKOrr2dUM7B2ityv9te7KB7Xq8/export?gid=0&format=tsv [following]\n",
      "--2020-04-15 09:23:57--  https://docs.google.com/spreadsheets/d/15HpBuwlUbm6Yg12qOtKOrr2dUM7B2ityv9te7KB7Xq8/export?gid=0&format=tsv\n",
      "Resolving docs.google.com (docs.google.com)... 216.58.209.14, 2a00:1450:401b:808::200e\n",
      "Connecting to docs.google.com (docs.google.com)|216.58.209.14|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: unspecified [text/tab-separated-values]\n",
      "Saving to: ‘/home/user/qiime2-moving-pictures-tutorial//sample-metadata.tsv’\n",
      "\n",
      "/home/user/qiime2-m     [ <=>                ]   2.04K  --.-KB/s    in 0s      \n",
      "\n",
      "2020-04-15 09:23:58 (14.0 MB/s) - ‘/home/user/qiime2-moving-pictures-tutorial//sample-metadata.tsv’ saved [2094]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget -O $workdir/\"sample-metadata.tsv\" \\\n",
    "  \"https://data.qiime2.org/2020.2/tutorials/moving-pictures/sample_metadata.tsv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, download the demultiplexed sequences that we’ll use in this analysis. To learn how to start a QIIME 2 analysis from fastq-formatted sequence data, see the [importing data tutorial](https://docs.qiime2.org/2020.2/tutorials/importing/). \n",
    "\n",
    "\n",
    "You can choose either a 1% subsample of the reads or a 10% subsample of the reads. If you’re just trying to gain experience with preparing and combining multiple sequencing runs of data, you can work with the 1% subsample data so that the commands will run very quickly. \n",
    "\n",
    "We’ll need to download two sets of demultiplexed sequences, each corresponding to one of the sequencing runs.\n",
    "If you’re using this tutorial to gain additional experience in generating and interpreting QIIME 2 analysis results, you should work with the 10% subsample data so that the results will be supported by more sequence data (1% of the reads is likely not sufficient to support some of the findings of the original study)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 10% subsample data\n",
    "\n",
    "Now download the demultiplexed sequences that we’ll use in this analysis. The following two commands will do it for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-04-15 09:24:05--  https://data.qiime2.org/2020.2/tutorials/fmt/fmt-tutorial-demux-1-10p.qza\n",
      "Resolving data.qiime2.org (data.qiime2.org)... 52.35.38.247\n",
      "Connecting to data.qiime2.org (data.qiime2.org)|52.35.38.247|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 FOUND\n",
      "Location: https://s3-us-west-2.amazonaws.com/qiime2-data/2020.2/tutorials/fmt/fmt-tutorial-demux-1-10p.qza [following]\n",
      "--2020-04-15 09:24:06--  https://s3-us-west-2.amazonaws.com/qiime2-data/2020.2/tutorials/fmt/fmt-tutorial-demux-1-10p.qza\n",
      "Resolving s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)... 52.218.201.224\n",
      "Connecting to s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)|52.218.201.224|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 20536136 (20M) [binary/octet-stream]\n",
      "Saving to: ‘/home/user/qiime2-moving-pictures-tutorial//fmt-tutorial-demux-1.qza’\n",
      "\n",
      "/home/user/qiime2-m 100%[===================>]  19.58M   452KB/s    in 55s     \n",
      "\n",
      "2020-04-15 09:25:02 (361 KB/s) - ‘/home/user/qiime2-moving-pictures-tutorial//fmt-tutorial-demux-1.qza’ saved [20536136/20536136]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget -O $workdir/\"fmt-tutorial-demux-1.qza\" \\\n",
    "  \"https://data.qiime2.org/2020.2/tutorials/fmt/fmt-tutorial-demux-1-10p.qza\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-03-18 10:55:32--  https://data.qiime2.org/2020.2/tutorials/fmt/fmt-tutorial-demux-2-10p.qza\n",
      "Resolving data.qiime2.org (data.qiime2.org)... 52.35.38.247\n",
      "Connecting to data.qiime2.org (data.qiime2.org)|52.35.38.247|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 FOUND\n",
      "Location: https://s3-us-west-2.amazonaws.com/qiime2-data/2020.2/tutorials/fmt/fmt-tutorial-demux-2-10p.qza [following]\n",
      "--2020-03-18 10:55:33--  https://s3-us-west-2.amazonaws.com/qiime2-data/2020.2/tutorials/fmt/fmt-tutorial-demux-2-10p.qza\n",
      "Resolving s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)... 52.218.212.120\n",
      "Connecting to s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)|52.218.212.120|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 8633726 (8.2M) [binary/octet-stream]\n",
      "Saving to: ‘fmt-tutorial-demux-2.qza’\n",
      "\n",
      "fmt-tutorial-demux- 100%[===================>]   8.23M  1.26MB/s    in 7.3s    \n",
      "\n",
      "2020-03-18 10:55:41 (1.12 MB/s) - ‘fmt-tutorial-demux-2.qza’ saved [8633726/8633726]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget -O $workdir/\"fmt-tutorial-demux-2.qza\" \\\n",
    "  \"https://data.qiime2.org/2020.2/tutorials/fmt/fmt-tutorial-demux-2-10p.qza\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1% subsample data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-03-18 10:55:41--  https://data.qiime2.org/2020.2/tutorials/fmt/fmt-tutorial-demux-1-1p.qza\n",
      "Resolving data.qiime2.org (data.qiime2.org)... 52.35.38.247\n",
      "Connecting to data.qiime2.org (data.qiime2.org)|52.35.38.247|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 FOUND\n",
      "Location: https://s3-us-west-2.amazonaws.com/qiime2-data/2020.2/tutorials/fmt/fmt-tutorial-demux-1-1p.qza [following]\n",
      "--2020-03-18 10:55:42--  https://s3-us-west-2.amazonaws.com/qiime2-data/2020.2/tutorials/fmt/fmt-tutorial-demux-1-1p.qza\n",
      "Resolving s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)... 52.218.250.32\n",
      "Connecting to s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)|52.218.250.32|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2165499 (2.1M) [binary/octet-stream]\n",
      "Saving to: ‘fmt-tutorial-demux-1.qza’\n",
      "\n",
      "fmt-tutorial-demux- 100%[===================>]   2.06M   474KB/s    in 4.6s    \n",
      "\n",
      "2020-03-18 10:55:48 (458 KB/s) - ‘fmt-tutorial-demux-1.qza’ saved [2165499/2165499]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget -O $workdir/\"fmt-tutorial-demux-1.qza\" \\\n",
    "  \"https://data.qiime2.org/2020.2/tutorials/fmt/fmt-tutorial-demux-1-1p.qza\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-04-15 09:31:04--  https://data.qiime2.org/2020.2/tutorials/fmt/fmt-tutorial-demux-2-1p.qza\n",
      "Resolving data.qiime2.org (data.qiime2.org)... 52.35.38.247\n",
      "Connecting to data.qiime2.org (data.qiime2.org)|52.35.38.247|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 FOUND\n",
      "Location: https://s3-us-west-2.amazonaws.com/qiime2-data/2020.2/tutorials/fmt/fmt-tutorial-demux-2-1p.qza [following]\n",
      "--2020-04-15 09:31:05--  https://s3-us-west-2.amazonaws.com/qiime2-data/2020.2/tutorials/fmt/fmt-tutorial-demux-2-1p.qza\n",
      "Resolving s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)... 52.218.216.56\n",
      "Connecting to s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)|52.218.216.56|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 935392 (913K) [binary/octet-stream]\n",
      "Saving to: ‘/home/user/qiime2-moving-pictures-tutorial//fmt-tutorial-demux-2.qza’\n",
      "\n",
      "/home/user/qiime2-m 100%[===================>] 913.47K   490KB/s    in 1.9s    \n",
      "\n",
      "2020-04-15 09:31:07 (490 KB/s) - ‘/home/user/qiime2-moving-pictures-tutorial//fmt-tutorial-demux-2.qza’ saved [935392/935392]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget -O $workdir/\"fmt-tutorial-demux-2.qza\" \\\n",
    "  \"https://data.qiime2.org/2020.2/tutorials/fmt/fmt-tutorial-demux-2-1p.qza\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Importing data as a qiime2 artifact\n",
    "\n",
    "All data that is used as input to QIIME 2 is in form of QIIME 2 artifacts, which contain information about the type of data and the source of the data. The first thing we need to do is import these sequence data files into a QIIME 2 artifact.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_metadata = qiime2.Metadata.load(workdir+'/sample-metadata.tsv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "fmt_tutorial_demux_1 = qiime2.Artifact.load(workdir+'/fmt-tutorial-demux-1.qza')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "fmt_tutorial_demux_2 = qiime2.Artifact.load(workdir+'/fmt-tutorial-demux-2.qza')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sequence quality control\n",
    "\n",
    "We’ll begin by performing quality control on the demultiplexed sequences using [DADA2](https://www.ncbi.nlm.nih.gov/pubmed/27214047) , but this time we’ll run the `denoise-single` command on each set of demultiplexed sequences individually. Again, we’ll want to start by visualizing sequence quality for some of the samples in each run. When we run denoise-single, we need to use the same values for `--p-trunc-len` and `--p-trim-left` for both runs, so when looking at the visualizations that result from these two commands, think about what values would make sense for these parameters for both commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "demux_summary_1 = demux.visualizers.summarize(fmt_tutorial_demux_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "demux_summary_2 = demux.visualizers.summarize(fmt_tutorial_demux_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output visualizations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><img onload=\"(function(div, url){\n",
       "if (typeof require !== 'undefined') {\n",
       "    var baseURL = require.toUrl('').split('/').slice(0, -2).join('/');\n",
       "} else {\n",
       "    var baseURL = JSON.parse(\n",
       "        document.getElementById('jupyter-config-data').innerHTML\n",
       "    ).baseUrl.slice(0, -1);\n",
       "}\n",
       "url = baseURL + url;\n",
       "fetch(url).then(function(res) {\n",
       "    if (res.status === 404) {\n",
       "        div.innerHTML = 'Install QIIME 2 Jupyter extension with:<br />' +\n",
       "                        '<code>jupyter serverextension enable --py qiime2' +\n",
       "                        ' --sys-prefix</code><br />then restart your server.' +\n",
       "                        '<br /><br />(Interactive output not available on ' +\n",
       "                        'static notebook viewer services like nbviewer.)';\n",
       "    } else if (res.status === 409) {\n",
       "        div.innerHTML = 'Visualization no longer in scope. Re-run this cell' +\n",
       "                        ' to see the visualization.';\n",
       "    } else if (res.ok) {\n",
       "        url = res.url;\n",
       "        div.innerHTML = '<iframe src=\\'' + url + '\\' style=\\'' +\n",
       "                        'width: 100%; height: 700px; border: 0;\\'>' +\n",
       "                        '</iframe><hr />Open in a: <a href=\\'' + url + '\\'' +\n",
       "                        ' target=\\'_blank\\'>new window</a>'\n",
       "    } else {\n",
       "        div.innerHTML = 'Something has gone wrong. Check notebook server for' +\n",
       "                        ' errors.';\n",
       "    }\n",
       "});\n",
       "})(this.parentElement, '/qiime2/redirect?location=/tmp/qiime2-archive-6_8yafpr')\" src=\"data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==\" /></div>"
      ],
      "text/plain": [
       "<visualization: Visualization uuid: a429c3fa-ecb3-4a80-a37a-eaae9a40f5ba>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "demux_summary_1.visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><img onload=\"(function(div, url){\n",
       "if (typeof require !== 'undefined') {\n",
       "    var baseURL = require.toUrl('').split('/').slice(0, -2).join('/');\n",
       "} else {\n",
       "    var baseURL = JSON.parse(\n",
       "        document.getElementById('jupyter-config-data').innerHTML\n",
       "    ).baseUrl.slice(0, -1);\n",
       "}\n",
       "url = baseURL + url;\n",
       "fetch(url).then(function(res) {\n",
       "    if (res.status === 404) {\n",
       "        div.innerHTML = 'Install QIIME 2 Jupyter extension with:<br />' +\n",
       "                        '<code>jupyter serverextension enable --py qiime2' +\n",
       "                        ' --sys-prefix</code><br />then restart your server.' +\n",
       "                        '<br /><br />(Interactive output not available on ' +\n",
       "                        'static notebook viewer services like nbviewer.)';\n",
       "    } else if (res.status === 409) {\n",
       "        div.innerHTML = 'Visualization no longer in scope. Re-run this cell' +\n",
       "                        ' to see the visualization.';\n",
       "    } else if (res.ok) {\n",
       "        url = res.url;\n",
       "        div.innerHTML = '<iframe src=\\'' + url + '\\' style=\\'' +\n",
       "                        'width: 100%; height: 700px; border: 0;\\'>' +\n",
       "                        '</iframe><hr />Open in a: <a href=\\'' + url + '\\'' +\n",
       "                        ' target=\\'_blank\\'>new window</a>'\n",
       "    } else {\n",
       "        div.innerHTML = 'Something has gone wrong. Check notebook server for' +\n",
       "                        ' errors.';\n",
       "    }\n",
       "});\n",
       "})(this.parentElement, '/qiime2/redirect?location=/tmp/qiime2-archive-g41fi758')\" src=\"data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==\" /></div>"
      ],
      "text/plain": [
       "<visualization: Visualization uuid: e66b01e4-da02-4a82-bc9a-6c53f69482a7>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "demux_summary_2.visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question\n",
    "\n",
    "Based on the plots you see in `demux-summary-1.qzv` and `demux-summary-2.qzv`, what values would you choose for `trunc_len` and `trim_left` in this case? How does these plots compare to those generated in the the moving pictures tutorial?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here the quality seems relatively low in the first few bases, and then seems to stay relatively high through the end of the reads. We’ll therefore trim the first 13 bases from each sequence and truncate the sequences at 150 bases. Since the reads are 151 bases long, this results in very little truncation of the sequences.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running external command line application(s). This may print messages to stdout and/or stderr.\n",
      "The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.\n",
      "\n",
      "Command: run_dada_single.R /tmp/qiime2-archive-dwefcyu1/5db90b62-c6e4-4c09-8c79-c0cdfbe2cea0/data /tmp/tmpg2t0es41/output.tsv.biom /tmp/tmpg2t0es41/track.tsv /tmp/tmpg2t0es41 150 13 2.0 2 Inf consensus 1.0 1 1000000 NULL 16\n",
      "\n"
     ]
    }
   ],
   "source": [
    "denoised_sequences_1 = dada2.methods.denoise_single(fmt_tutorial_demux_1, trim_left = 13, trunc_len = 150)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running external command line application(s). This may print messages to stdout and/or stderr.\n",
      "The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.\n",
      "\n",
      "Command: run_dada_single.R /tmp/qiime2-archive-zvr04t4i/e7ae20b7-0d6c-4251-a0d6-2fa6463e11f1/data /tmp/tmp4l27d8ri/output.tsv.biom /tmp/tmp4l27d8ri/track.tsv /tmp/tmp4l27d8ri 150 13 2.0 2 Inf consensus 1.0 1 1000000 NULL 16\n",
      "\n"
     ]
    }
   ],
   "source": [
    "denoised_sequences_2 = dada2.methods.denoise_single(fmt_tutorial_demux_2, trim_left = 13, trunc_len = 150)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Viewing denoising stats\n",
    "\n",
    "The `denoise_single` commands return basic statistics about the denoising process, and can be visualized with the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><img onload=\"(function(div, url){\n",
       "if (typeof require !== 'undefined') {\n",
       "    var baseURL = require.toUrl('').split('/').slice(0, -2).join('/');\n",
       "} else {\n",
       "    var baseURL = JSON.parse(\n",
       "        document.getElementById('jupyter-config-data').innerHTML\n",
       "    ).baseUrl.slice(0, -1);\n",
       "}\n",
       "url = baseURL + url;\n",
       "fetch(url).then(function(res) {\n",
       "    if (res.status === 404) {\n",
       "        div.innerHTML = 'Install QIIME 2 Jupyter extension with:<br />' +\n",
       "                        '<code>jupyter serverextension enable --py qiime2' +\n",
       "                        ' --sys-prefix</code><br />then restart your server.' +\n",
       "                        '<br /><br />(Interactive output not available on ' +\n",
       "                        'static notebook viewer services like nbviewer.)';\n",
       "    } else if (res.status === 409) {\n",
       "        div.innerHTML = 'Visualization no longer in scope. Re-run this cell' +\n",
       "                        ' to see the visualization.';\n",
       "    } else if (res.ok) {\n",
       "        url = res.url;\n",
       "        div.innerHTML = '<iframe src=\\'' + url + '\\' style=\\'' +\n",
       "                        'width: 100%; height: 700px; border: 0;\\'>' +\n",
       "                        '</iframe><hr />Open in a: <a href=\\'' + url + '\\'' +\n",
       "                        ' target=\\'_blank\\'>new window</a>'\n",
       "    } else {\n",
       "        div.innerHTML = 'Something has gone wrong. Check notebook server for' +\n",
       "                        ' errors.';\n",
       "    }\n",
       "});\n",
       "})(this.parentElement, '/qiime2/redirect?location=/tmp/qiime2-archive-civ2dffr')\" src=\"data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==\" /></div>"
      ],
      "text/plain": [
       "<visualization: Visualization uuid: cae9ddc3-4822-4f24-b9fd-fe86fdcd7bff>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "denoise_stats_1 = metadata.visualizers.tabulate(denoised_sequences_1.denoising_stats.view(qiime2.Metadata))\n",
    "denoise_stats.visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><img onload=\"(function(div, url){\n",
       "if (typeof require !== 'undefined') {\n",
       "    var baseURL = require.toUrl('').split('/').slice(0, -2).join('/');\n",
       "} else {\n",
       "    var baseURL = JSON.parse(\n",
       "        document.getElementById('jupyter-config-data').innerHTML\n",
       "    ).baseUrl.slice(0, -1);\n",
       "}\n",
       "url = baseURL + url;\n",
       "fetch(url).then(function(res) {\n",
       "    if (res.status === 404) {\n",
       "        div.innerHTML = 'Install QIIME 2 Jupyter extension with:<br />' +\n",
       "                        '<code>jupyter serverextension enable --py qiime2' +\n",
       "                        ' --sys-prefix</code><br />then restart your server.' +\n",
       "                        '<br /><br />(Interactive output not available on ' +\n",
       "                        'static notebook viewer services like nbviewer.)';\n",
       "    } else if (res.status === 409) {\n",
       "        div.innerHTML = 'Visualization no longer in scope. Re-run this cell' +\n",
       "                        ' to see the visualization.';\n",
       "    } else if (res.ok) {\n",
       "        url = res.url;\n",
       "        div.innerHTML = '<iframe src=\\'' + url + '\\' style=\\'' +\n",
       "                        'width: 100%; height: 700px; border: 0;\\'>' +\n",
       "                        '</iframe><hr />Open in a: <a href=\\'' + url + '\\'' +\n",
       "                        ' target=\\'_blank\\'>new window</a>'\n",
       "    } else {\n",
       "        div.innerHTML = 'Something has gone wrong. Check notebook server for' +\n",
       "                        ' errors.';\n",
       "    }\n",
       "});\n",
       "})(this.parentElement, '/qiime2/redirect?location=/tmp/qiime2-archive-tqszp80d')\" src=\"data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==\" /></div>"
      ],
      "text/plain": [
       "<visualization: Visualization uuid: bc59dcff-1db6-4d89-9d16-e174676ad9af>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "denoise_stats_2 = metadata.visualizers.tabulate(denoised_sequences_2.denoising_stats.view(qiime2.Metadata))\n",
    "denoise_stats.visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Merging denoised data\n",
    "\n",
    "The `denoise_single` command is the last step in this analysis that needs to be run on a per sequencing run basis. We’re therefore ready to merge the artifacts generated by those two commands. First we’ll merge the two `FeatureTable[Frequency]` artifacts, and then we’ll merge the two `FeatureData[Sequence`] artifacts. This is possible because the feature ids generated in each run of `denoise_single` are directly comparable (in this case, the feature id is the md5 hash of the sequence defining the feature)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "merged_table = feature_table.methods.merge([denoised_sequences_1.table,\n",
    "                                            denoised_sequences_2.table])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "rep_seqs = feature_table.methods.merge_seqs([denoised_sequences_1.representative_sequences,\n",
    "                                             denoised_sequences_2.representative_sequences])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we’ll generate a summary of the merged `FeatureTable[Frequency]` artifact."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'feature_table' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-55190f02b63d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mvisualization_table\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfeature_table\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvisualizers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msummarize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmerged_table\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmerged_table\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msample_metadata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mvisualization_table\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvisualization\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'feature_table' is not defined"
     ]
    }
   ],
   "source": [
    "visualization_table = feature_table.visualizers.summarize(merged_table.merged_table, sample_metadata)\n",
    "visualization_table.visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question\n",
    "\n",
    "> Based on the information in `table.qzv`, what value will you choose for the `--p-sampling-depth` parameter when you run `qiime diversity core-metrics-phylogenetic`?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question\n",
    "\n",
    "> Generate summaries of the tables for the individual runs of `qiime dada2 denoise-single`. How many features were defined in the first run? How many features were defined in the second run? How do these numbers compare to total number of features after merging?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We’ll also generate a summary of the merged FeatureData[Sequence] artifact. You can use this summary to obtain additional information about specific features of interest as you proceed through the analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><img onload=\"(function(div, url){\n",
       "if (typeof require !== 'undefined') {\n",
       "    var baseURL = require.toUrl('').split('/').slice(0, -2).join('/');\n",
       "} else {\n",
       "    var baseURL = JSON.parse(\n",
       "        document.getElementById('jupyter-config-data').innerHTML\n",
       "    ).baseUrl.slice(0, -1);\n",
       "}\n",
       "url = baseURL + url;\n",
       "fetch(url).then(function(res) {\n",
       "    if (res.status === 404) {\n",
       "        div.innerHTML = 'Install QIIME 2 Jupyter extension with:<br />' +\n",
       "                        '<code>jupyter serverextension enable --py qiime2' +\n",
       "                        ' --sys-prefix</code><br />then restart your server.' +\n",
       "                        '<br /><br />(Interactive output not available on ' +\n",
       "                        'static notebook viewer services like nbviewer.)';\n",
       "    } else if (res.status === 409) {\n",
       "        div.innerHTML = 'Visualization no longer in scope. Re-run this cell' +\n",
       "                        ' to see the visualization.';\n",
       "    } else if (res.ok) {\n",
       "        url = res.url;\n",
       "        div.innerHTML = '<iframe src=\\'' + url + '\\' style=\\'' +\n",
       "                        'width: 100%; height: 700px; border: 0;\\'>' +\n",
       "                        '</iframe><hr />Open in a: <a href=\\'' + url + '\\'' +\n",
       "                        ' target=\\'_blank\\'>new window</a>'\n",
       "    } else {\n",
       "        div.innerHTML = 'Something has gone wrong. Check notebook server for' +\n",
       "                        ' errors.';\n",
       "    }\n",
       "});\n",
       "})(this.parentElement, '/qiime2/redirect?location=/tmp/qiime2-archive-5x0pm_ww')\" src=\"data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==\" /></div>"
      ],
      "text/plain": [
       "<visualization: Visualization uuid: 3219db90-d5bf-46ee-80c3-cdfa2eab3745>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rep_seq_viz = feature_table.visualizers.tabulate_seqs(rep_seqs.merged_data)\n",
    "rep_seq_viz.visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Saving visualisation locally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/user/qiime2-moving-pictures-tutorial//rep-seqs.qzv'"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rep_seq_viz.visualization.save(workdir+'/rep-seqs.qzv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Diversity analysis\n",
    "\n",
    "Now that you have `FeatureTable[Frequency]` and `FeatureData[Sequence]` objects, you’re ready to begin exploring the composition of these samples in the context of their metadata. Refer to the [moving pictures](https://docs.qiime2.org/2020.2/tutorials/moving-pictures/) tutorial to derive the specific commands that you’ll run. Several questions concern longitudinal changes in the microbiome of individuals; review the actions described in the [q2-longitudinal tutorial](https://docs.qiime2.org/2020.2/tutorials/longitudinal/) to learn about methods for longitudinal analysis that are supported in QIIME 2.\n",
    "\n",
    "Below are some specific questions to answer about this data, grouped into a few categories. Try to collect at least one specific result to support your answer to each question.\n",
    "\n",
    "1. The personal human microbiome.\n",
    "    1. Do samples differ in composition by subject-id (i.e., across individual)?\n",
    "    2. Do samples differ in richness by subject-id?\n",
    "    3. Do samples differ in evenness by subject-id?\n",
    "    4. Do richness, evenness, composition, and UniFrac distance change in individuals between baseline and the end of the study? Does this differ between individuals receiving FMT and control subjects? (Hint: try the paired difference/distance methods described in the [q2-longitudinal tutorial](https://docs.qiime2.org/2020.2/tutorials/longitudinal/)\n",
    "    5. Do richness, evenness, composition, and UniFrac distance change over time and in relation to FMT treatment and other subject metadata? Are these metrics more variable over time in treatment or control groups? (Hint: these questions concern longitudinal measurements.)\n",
    "\n",
    "2. Microbiota engraftment. \n",
    "    1. At approximately what week in the study do microbiome samples in individuals who receive treatment appear most similar to FMT donors in terms of unweighted UniFrac distances? (Hint: Try plotting the data with `qiime emperor plot`. Pay close attention to the color tab and visibility menu.) \n",
    "    2. At approximately what week in the study do microbiome samples in individuals who receive treatment appear most similar to FMT donors in terms of Bray-Curtis distances? \n",
    "    3. Is this pattern stronger based on unweighted UniFrac or Bray-Curtis distance? Based on how you know about these metrics, what does this suggest to you about what is changing in the microbiome with fecal microbiota transplant? Use the Jaccard and weighted UniFrac distance Emperor plots to help you refine this idea.\n",
    "\n",
    "3. Experimental design: Comparing stool and swab sample collection methods.\n",
    "    1. What feature(s) differ most in abundance between the stool and swab samples? What taxonomy is associated with those feature ids based on their best BLAST hits, and based on the results of Naive Bayes feature classification with the QIIME 2 `q2-feature-classifier` plugin?\n",
    "    2. Is the microbial composition of stool and swab samples significantly different based on either unweighted UniFrac or Bray-Curtis distances between samples (yes, no, or not possible to say with the current information)?\n",
    "    3. Do the donated fecal material samples appear more similar in composition to the stool or swab samples?\n",
    "    4. Does community richness differ between stool samples and swab samples? Does community evenness differ between stool samples and swab samples?\n",
    "\n",
    "4. How many samples were sequenced in each sequencing run? Do you observe any systematic differences in the samples across sequencing runs?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Acknowledgements\n",
    "\n",
    "The data in this tutorial was initially presented in: Microbiota Transfer Therapy alters gut ecosystem and improves gastrointestinal and autism symptoms: an open-label study. Dae-Wook Kang, James B. Adams, Ann C. Gregory, Thomas Borody, Lauren Chittick, Alessio Fasano, Alexander Khoruts, Elizabeth Geis, Juan Maldonado, Sharon McDonough-Means, Elena L. Pollard, Simon Roux, Michael J. Sadowsky, Karen Schwarzberg Lipson, Matthew B. Sullivan, J. Gregory Caporaso and Rosa Krajmalnik-Brown. Microbiome (2017) 5:10. DOI: 10.1186/s40168-016-0225-7."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}