Merge pull request #11 from zbyosufzai/patch-1

Updated Mamba and Nextflow
NIGMS · Mar 21, 2024 · edcc766 · edcc766
2 parents 06a2e1c + 4afa25b
commit edcc766
Show file tree

Hide file tree

Showing 7 changed files with 326 additions and 138 deletions.
diff --git a/docs/human_sperm.config b/docs/human_sperm.config
@@ -3,10 +3,9 @@ profiles{
       process.executor = 'google-batch'
       google.location = 'us-central1'
       google.region  = 'us-central1'
-      google.project = 'nosi-hawaii-dna-27fa'
+      google.project = '<PROJECT_ID>'
       workDir = "gs://dna-methyl/methyseq_sperm/work"
       params.outdir = "gs://dna-methyl/methyseq_sperm/results"
-      params.input = 'Tutorial_4/sra_download/*_{1,2}.fastq'
       process.machineType = 'c2-standard-16'
      }
 }
diff --git a/docs/test_LS.config b/docs/test_LS.config
@@ -10,7 +10,7 @@ profiles{
       google {
           location = 'us-central1'
           region  = 'us-central1'
-          project = 'nosi-hawaii-dna-27fa'
+          project = '<Project_ID>'
       }
       params {
           outdir = 'gs://demo-nextflow-bucket/test/output'

diff --git a/tutorial_1-bismark.ipynb b/tutorial_1-bismark.ipynb
@@ -53,19 +53,22 @@
   {
    "cell_type": "markdown",
    "id": "a9b2aaca-ac15-4596-9af6-fc05d0df2dd2",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "source": [
-    "**Note**: if conda is NOT installed, we need to install conda first by using the following commands:\n",
+    "<div class=\"alert alert-block alert-success\"><b>Note</b>: if conda is NOT installed, we need to install conda first by using the following commands:\n",
+    "    \n",
     "```bash\n",
     "! mkdir -p ~/miniconda3\n",
     "! wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh\n",
     "! bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3\n",
     "! rm -rf ~/miniconda3/miniconda.sh\n",
     "! ~/miniconda3/bin/conda init bash\n",
     "! ~/miniconda3/bin/conda init zsh\n",
-    "```\n",
-    "\n",
-    "It would take less time to install some of the tools in this tutorial if you create a new conda environment. But it is a little bit tricky to use a different conda environment within Jupyter in GCP. You can find instructions of how to create a new conda environment [here](https://github.com/STRIDES/NIHCloudLabGCP/docs/How_to_use_conda_envs_as_kernels.ipynb)."
+    "```   \n",
+    "It would take less time to install some of the tools in this tutorial if you create a new conda environment. But it is a little bit tricky to use a different conda environment within Jupyter in GCP. You can find instructions of how to create a new conda environment [here](https://github.com/STRIDES/NIHCloudLabGCP/docs/How_to_use_conda_envs_as_kernels.ipynb).\n",
+    "</div>"
    ]
   },
   {
@@ -75,10 +78,10 @@
    "source": [
     "#### Mamba \n",
     "\n",
-    "[**mamba**](https://mamba.readthedocs.io/en/latest/user_guide/mamba.html) is a re-implementation of the conda package manager in C++. It uses the same commands and configuration options as conda . The only difference is that you should still use conda for activation and deactivation. Once conda is installed, we will install `mamba` and use it to install all the tools we are going to use in Tutorial 1 and 2. \n",
+    "[**mamba**](https://mamba.readthedocs.io/en/latest/user_guide/mamba.html) is a re-implementation of the conda package manager in C++. It uses the same commands and configuration options as conda. The only difference is that you should still use conda for activation and deactivation. Once conda is installed, we will install `mamba` and use it to install all the tools we are going to use in Tutorial 1 and 2. \n",
     "> **Installation**: conda -> mamba -> other tools\n",
     "\n",
-    "The script below installs `mamba` into the 'base' environment from the conda-forge channel. The `-y` flag is used to Sets any confirmation values to 'yes' automatically."
+    "The script below installs `mamba` via Mambaforge."
    ]
   },
   {
@@ -91,7 +94,22 @@
    },
    "outputs": [],
    "source": [
-    "! conda install mamba -n base -c conda-forge -y"
+    "! curl -L -O https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh\n",
+    "! bash Mambaforge-$(uname)-$(uname -m).sh -b -p $HOME/mambaforge"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "59a0266e-9afa-4695-af55-ece62a3713be",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "#This need to be run at the start of every notebook\n",
+    "import os\n",
+    "os.environ[\"PATH\"] += os.pathsep + os.environ[\"HOME\"]+\"/mambaforge/bin\""
    ]
   },
   {
@@ -161,7 +179,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "775b7955-f05f-4334-9192-807cf464617f",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "# Show current working directory\n",
@@ -234,7 +254,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "dc94be73-c228-4021-b22f-8e3590f88901",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "! gsutil cp gs://nigms-sandbox/dna-methyl/Mus_musculus.GRCm39.dna.chromosome.6.fa Tutorial_1/ref_genome"
@@ -281,7 +303,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "e00a553e-8115-4d52-a9db-9d7c2b5cbcf5",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "from IPython.display import IFrame\n",
@@ -499,7 +523,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "5f789349-40ec-4d25-b308-c2772094c549",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "! zcat Tutorial_1/bismark/SRX202087_R1_val_1_bismark_bt2_pe.deduplicated.bedGraph.gz | head -n 10"
@@ -509,7 +535,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "ff63dfc6-e0c2-4210-99ed-edd100dab334",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "! zcat Tutorial_1/bismark/SRX202087_R1_val_1_bismark_bt2_pe.deduplicated.bismark.cov.gz | head -n 10"
@@ -599,7 +627,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "9cbdcef8-9571-49b3-a540-fd75172bfc02",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "! pip install igv-notebook"
@@ -619,7 +649,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "32d5808b-91d7-4d37-88cb-3914f2ae8a20",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "import igv_notebook\n",
@@ -694,7 +726,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "24117698-6013-4a55-bd91-dea1a7f8c61e",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "b2 = igv_notebook.Browser(\n",
@@ -790,7 +824,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "2c730ddb-5629-4b6d-b587-29978f688212",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "# Zoom in to see a specific region in the above browser\n",
@@ -820,7 +856,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "3c0f4e59-ffb3-4f80-9adf-ded3e75b4371",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "!pip install jupytercards --quiet\n",
@@ -832,7 +870,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "bf2d31cf-03e3-4ef6-970b-8f2ce5446cc9",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "!pip install jupyterquiz==2.0.7 --quiet\n",
@@ -876,15 +916,15 @@
  ],
  "metadata": {
   "environment": {
-   "kernel": "python3",
-   "name": "common-cpu.m103",
+   "kernel": "conda-root-py",
+   "name": "workbench-notebooks.m118",
    "type": "gcloud",
-   "uri": "gcr.io/deeplearning-platform-release/base-cpu:m103"
+   "uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m118"
   },
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel) (Local)",
    "language": "python",
-   "name": "python3"
+   "name": "conda-root-py"
   },
   "language_info": {
    "codemirror_mode": {
@@ -896,7 +936,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.12"
+   "version": "3.10.13"
   }
  },
  "nbformat": 4,

diff --git a/tutorial_2-metilene.ipynb b/tutorial_2-metilene.ipynb
@@ -60,7 +60,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "946bfe1e-7e51-4b45-98be-774f6ea48f4b",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "# Create a directory called Tutorial_2\n",
@@ -84,7 +86,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "9ddef02c-cc44-43a1-b6aa-36c5a6248de9",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "# Sort all the methylation files\n",
@@ -110,9 +114,14 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "d0add508-27f3-40ce-a223-e36266c8693c",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
+    "import os\n",
+    "os.environ[\"PATH\"] += os.pathsep + os.environ[\"HOME\"]+\"/mambaforge/bin\"\n",
+    "\n",
     "! metilene_input.pl \\\n",
     "    --in1 Tutorial_2/SRX202087.bedgraph,Tutorial_2/SRX271141.bedgraph \\\n",
     "    --in2 Tutorial_2/SRX202088.bedgraph,Tutorial_2/SRX271142.bedgraph \\\n",
@@ -133,7 +142,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "6f9a75b6-5a4b-43f3-8b5b-64334e1ec40d",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "! head -30 Tutorial_2/metilene.input "
@@ -239,7 +250,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "e338c101-c784-4058-ba17-6ec73375b18c",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "!metilene_output.pl -q Tutorial_2/metilene_denovo.output -o Tutorial_2/denovo -a serum -b 2i\n",
@@ -368,7 +381,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "9947f039-5280-4a07-9097-f15a978e7eb7",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "!pip install jupytercards --quiet\n",
@@ -380,7 +395,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "3abc5c18-e21d-4c98-b666-198d622c4d0d",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "!pip install jupyterquiz==2.0.7 --quiet\n",
@@ -401,15 +418,15 @@
  ],
  "metadata": {
   "environment": {
-   "kernel": "python3",
-   "name": "common-cpu.m103",
+   "kernel": "conda-root-py",
+   "name": "workbench-notebooks.m118",
    "type": "gcloud",
-   "uri": "gcr.io/deeplearning-platform-release/base-cpu:m103"
+   "uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m118"
   },
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel) (Local)",
    "language": "python",
-   "name": "python3"
+   "name": "conda-root-py"
   },
   "language_info": {
    "codemirror_mode": {
@@ -421,7 +438,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.12"
+   "version": "3.10.13"
   }
  },
  "nbformat": 4,