Bioconda #10
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,154 +1,34 @@ | ||
| FROM nfcore/base | ||
| MAINTAINER Phil Ewels <[email protected]> | ||
| LABEL authors="[email protected]" \ | ||
| description="Docker image containing all requirements for the nfcore/ChIPseq pipeline" | ||
|
|
||
| COPY environment.yml / | ||
| RUN conda update -n base conda | ||
| RUN conda update --all | ||
| RUN conda env update -n root --file environment.yml && conda clean -a | ||
|
There was a problem hiding this comment. This is brilliant - I didn't know that you could just use the root environment like this. Much cleaner 👍 |
||
| ENV PATH /opt/conda/envs/nfcore-chipseq-1.4dev/bin:$PATH | ||
|
There was a problem hiding this comment. If we're not using a conda environment, I guess we can probably get rid of this line? This was an alternative to There was a problem hiding this comment. I'm not very familiar with the conda environment. But If I understand it right we might be able to remove that line. It is better to check that before. I have a question about nexflow. Can I specify which process to reexecute (even after completed)? I removed the folder of the process in the work directory and it just restarted all processes. There was a problem hiding this comment.
No, it's automatic - if it thinks that downstream processes rely on a work folder that you deleted, then it will rerun them all. It does this because it can't know that the downstream results would be the same otherwise. If I'm misunderstanding your question, then come ask on gitter where we can chat: https://gitter.im/nf-core/Lobby |
||
|
|
||
| ENV NGSPLOT_VERSION="2.63" | ||
| RUN curl -fsSL https://github.com/shenlab-sinai/ngsplot/archive/${NGSPLOT_VERSION}.tar.gz -o /opt/ngsplot_${NGSPLOT_VERSION}.tar.gz && \ | ||
|
There was a problem hiding this comment. Thanks for this! Whilst it works, we have previously discussed that we should try to avoid direct installations into docker images like this. Instead, we hope to package missing tools into bioconda. The reason for this is that users who aren't using docker and singularity, but only using the bioconda environment will not be able to run the full pipeline. So if possible, for maximum portability, it's best to try to use bioconda for everything. Your thoughts on this are welcome. Do you think it's possible to add these to bioconda? Would you be willing / able to do so..? 😉 There was a problem hiding this comment. I'm not sure this package is in Bioconda: shenlab-sinai/ngsplot#74 But still, the ngsplot genomes needs to be installed (they are available in google drive). That was one of the problems I had before. I'm not sure that is possible with Bioconda. What if we added a script and run it from docker? And users would be able to execute it locally? There was a problem hiding this comment.
Ah yeah, I'd forgotten about this. No they can't (or rather, shouldn't) be added to bioconda. I'd argue that they shouldn't really be in the docker container either. It works, but it makes the container huge and is bloated for anyone not using these reference genomes. I think this falls into the same category as all references - can we not just ask the user to fetch them and specify their location in the same way as alignment genome references? We could also look into adding them to AWS-iGenomes or somthing. We have also been discussing an upcoming project to create a bioconda-like resource for large genome references such as this which would be the best solution (see https://gitter.im/nf-core/Lobby?at=5afd44f02df44c2d0634e430) Phil There was a problem hiding this comment. @apeltzer / @chuan-wang - any thoughts or previous discussion points that I've missed here? There was a problem hiding this comment. Sorry @tiagochst, I missed a bit..
Yeah, I guess.... I would consider this a last resort 😅 There was a problem hiding this comment. Sorry still on vacation and thus only iPad typing available. I agree that we shouldn’t package this in bio conda and keep it external of the Container for the reasons you mentioned. There was a problem hiding this comment. @apeltzer - you mean the references yeah? And package the tool scripts in bioconda? It could also be worth bringing this topic up in the bioconda community to see what they have to say.. https://gitter.im/bioconda/Lobby There was a problem hiding this comment. Yes, just the references . I can ask once I’m back - already made a todo |
||
| tar xvzf /opt/ngsplot_${NGSPLOT_VERSION}.tar.gz -C /opt/ && \ | ||
| rm /opt/ngsplot_${NGSPLOT_VERSION}.tar.gz | ||
| ENV PATH=${PATH}:/opt/ngsplot-${NGSPLOT_VERSION}/bin | ||
| ENV NGSPLOT=/opt/ngsplot-${NGSPLOT_VERSION}/ | ||
|
|
||
| RUN wget "https://drive.google.com/uc?export=download&id=0B5hDZ2BucCI6SURYWW5XdUxnbW8" -O ngsplotdb_hg19_75_3.00.tar.gz && \ | ||
| echo y | ngsplotdb.py install ngsplotdb_hg19_75_3.00.tar.gz && \ | ||
| rm -rf ngsplotdb_hg19_75_3.00.tar.gz && \ | ||
| wget "https://drive.google.com/uc?export=download&id=0B5hDZ2BucCI6S3E4dVprdlF2YW8" -O ngsplotdb_hg38_76_3.00.tar.gz && \ | ||
| echo y | ngsplotdb.py install ngsplotdb_hg38_76_3.00.tar.gz && \ | ||
| rm -rf ngsplotdb_hg38_76_3.00.tar.gz && \ | ||
| wget "https://drive.google.com/uc?export=download&id=0B5hDZ2BucCI6NXNzNjZveXdadU0" -O ngsplotdb_mm10_75_3.00.tar.gz && \ | ||
| echo y | ngsplotdb.py install ngsplotdb_mm10_75_3.00.tar.gz && \ | ||
| rm -rf ngsplotdb_mm10_75_3.00.tar.gz | ||
|
|
||
| RUN git clone https://github.com/kundajelab/phantompeakqualtools && \ | ||
| mv phantompeakqualtools /opt/ && \ | ||
| echo 'library(caTools)' | cat - /opt/phantompeakqualtools/run_spp.R > temp && mv temp /opt/phantompeakqualtools/run_spp.R && \ | ||
| chmod 755 /opt/phantompeakqualtools/* && \ | ||
| echo 'alias run_spp.R="Rscript /opt/phantompeakqualtools/run_spp.R"' >> ~/.bashrc | ||
| ENV PATH=${PATH}:/opt/phantompeakqualtools | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,150 @@ | ||
| FROM openjdk:8 | ||
|
|
||
| LABEL author="Phil Ewels" \ | ||
| description="Docker image containing all requirements for nf-core/ChIPseq pipeline" \ | ||
| maintainer="[email protected]" | ||
|
|
||
| # Install container-wide requrements gcc, pip, zlib, libssl, make, libncurses, fortran77, g++, R | ||
| RUN apt-get update && \ | ||
| apt-get install -y --no-install-recommends \ | ||
| g++ \ | ||
| gawk \ | ||
| gcc \ | ||
| gfortran \ | ||
| libboost-all-dev \ | ||
| libbz2-dev \ | ||
| libcurl4-openssl-dev \ | ||
| libdbd-mysql \ | ||
| libgsl0-dev \ | ||
| liblzma-dev \ | ||
| libmariadb-client-lgpl-dev \ | ||
| libncurses5-dev \ | ||
| libpcre3-dev \ | ||
| libreadline-dev \ | ||
| libssl-dev \ | ||
| libxml2-dev \ | ||
| make \ | ||
| python-dev \ | ||
| zlib1g-dev \ | ||
| && rm -rf /var/lib/apt/lists/* | ||
|
|
||
| # Use gawk instead of awk (needed for phantompeakqualtools) | ||
| RUN update-alternatives --set awk /usr/bin/gawk | ||
|
|
||
| # Install pip | ||
| RUN curl -fsSL https://bootstrap.pypa.io/get-pip.py -o /opt/get-pip.py && \ | ||
| python /opt/get-pip.py && \ | ||
| rm /opt/get-pip.py | ||
|
|
||
| # Install FastQC | ||
| ENV FASTQC_BIN="fastqc_v0.11.5.zip" | ||
| RUN curl -fsSL http://www.bioinformatics.babraham.ac.uk/projects/fastqc/$FASTQC_BIN -o /opt/$FASTQC_BIN && \ | ||
| unzip /opt/$FASTQC_BIN -d /opt/ && \ | ||
| chmod 755 /opt/FastQC/fastqc && \ | ||
| ln -s /opt/FastQC/fastqc /usr/local/bin/fastqc && \ | ||
| rm /opt/$FASTQC_BIN | ||
|
|
||
| # Install cutadapt | ||
| RUN pip install cutadapt | ||
|
|
||
| # Install TrimGalore | ||
| ENV TRIMGALORE_VERSION="0.4.5" | ||
| RUN curl -fsSL https://github.com/FelixKrueger/TrimGalore/archive/${TRIMGALORE_VERSION}.tar.gz -o /opt/trimgalore_${TRIMGALORE_VERSION}.tar.gz && \ | ||
| tar xvzf /opt/trimgalore_${TRIMGALORE_VERSION}.tar.gz -C /opt/ && \ | ||
| ln -s /opt/TrimGalore-${TRIMGALORE_VERSION}/trim_galore /usr/local/bin/trim_galore && \ | ||
| rm /opt/trimgalore_${TRIMGALORE_VERSION}.tar.gz | ||
|
|
||
| # Install BWA | ||
| ENV BWA_VERSION="0.7.15" | ||
| RUN curl -fsSL https://downloads.sourceforge.net/project/bio-bwa/bwa-${BWA_VERSION}.tar.bz2 -o /opt/bwa-${BWA_VERSION}.tar.bz2 && \ | ||
| tar xvjf /opt/bwa-${BWA_VERSION}.tar.bz2 -C /opt/ && \ | ||
| cd /opt/bwa-${BWA_VERSION};make && \ | ||
| ln -s /opt/bwa-${BWA_VERSION}/bwa /usr/local/bin/bwa && \ | ||
| rm /opt/bwa-${BWA_VERSION}.tar.bz2 | ||
|
|
||
| # Install SAMTools | ||
| ENV SAMTOOLS_VERSON="1.6" | ||
| RUN curl -fsSL https://github.com/samtools/samtools/releases/download/${SAMTOOLS_VERSON}/samtools-${SAMTOOLS_VERSON}.tar.bz2 -o /opt/samtools-${SAMTOOLS_VERSON}.tar.bz2 && \ | ||
| tar xvjf /opt/samtools-${SAMTOOLS_VERSON}.tar.bz2 -C /opt/ && \ | ||
| cd /opt/samtools-${SAMTOOLS_VERSON};make;make install && \ | ||
| rm /opt/samtools-${SAMTOOLS_VERSON}.tar.bz2 | ||
|
|
||
| # Install PicardTools | ||
| ENV PICARD_VERSION="picard-tools-2.0.1" | ||
| RUN curl -fsSL https://github.com/broadinstitute/picard/releases/download/2.0.1/${PICARD_VERSION}.zip -o /opt/${PICARD_VERSION}.zip && \ | ||
| unzip /opt/${PICARD_VERSION}.zip -d /opt/ && \ | ||
| rm /opt/${PICARD_VERSION}.zip | ||
| ENV PICARD_HOME /opt/${PICARD_VERSION} | ||
|
|
||
| # Install BEDTools | ||
| ENV BEDTOOLS_VERSION="bedtools-2.26.0" | ||
| RUN curl -fsSL https://github.com/arq5x/bedtools2/releases/download/v2.26.0/${BEDTOOLS_VERSION}.tar.gz -o /opt/${BEDTOOLS_VERSION}.tar.gz && \ | ||
| tar xvzf /opt/${BEDTOOLS_VERSION}.tar.gz -C /opt/ && \ | ||
| cd /opt/bedtools2; make && \ | ||
| cp /opt/bedtools2/bin/* /usr/local/bin/ && \ | ||
| rm /opt/${BEDTOOLS_VERSION}.tar.gz | ||
|
|
||
| # Install R | ||
| ENV R_VERSION="R-3.4.2" | ||
| RUN curl -fsSL https://cran.r-project.org/src/base/R-3/${R_VERSION}.tar.gz -o /opt/${R_VERSION}.tar.gz && \ | ||
| tar xvzf /opt/${R_VERSION}.tar.gz -C /opt/ && \ | ||
| cd /opt/${R_VERSION};./configure;make;make install && \ | ||
| rm /opt/${R_VERSION}.tar.gz | ||
|
|
||
| # Install core R dependencies | ||
| RUN echo "r <- getOption('repos'); r['CRAN'] <- 'https://ftp.acc.umu.se/mirror/CRAN/'; options(repos = r);" > ~/.Rprofile && \ | ||
| Rscript -e "install.packages('caTools',dependencies=TRUE)" && \ | ||
| Rscript -e "install.packages('snow',dependencies=TRUE)" && \ | ||
| Rscript -e "install.packages('doMC',dependencies=TRUE)" && \ | ||
| Rscript -e "install.packages('utils',dependencies=TRUE)" && \ | ||
| Rscript -e "install.packages('stringr',dependencies=TRUE)" && \ | ||
| Rscript -e "install.packages('markdown',dependencies=TRUE)" && \ | ||
| Rscript -e "install.packages('evaluate',dependencies=TRUE)" && \ | ||
| Rscript -e "install.packages('ggplot2',dependencies=TRUE)" && \ | ||
| Rscript -e "install.packages('knitr',dependencies=TRUE)" && \ | ||
| Rscript -e "install.packages('RMySQL',dependencies=TRUE)" | ||
|
|
||
| # Install R Bioconductor packages | ||
| RUN echo 'source("https://bioconductor.org/biocLite.R")' > /opt/packages.r && \ | ||
| echo 'biocLite()' >> /opt/packages.r && \ | ||
| echo 'biocLite(c("BSgenome", "Rsamtools", "ShortRead", "GenomicRanges", "GenomicFeatures", "ensembldb", "ChIPpeakAnno", "biomaRt", "rtracklayer", "BSgenome.Hsapiens.UCSC.hg19", "org.Hs.eg.db", "BSgenome.Mmusculus.UCSC.mm10", "org.Mm.eg.db"))' >> /opt/packages.r && \ | ||
| Rscript /opt/packages.r && \ | ||
| mkdir /usr/local/lib/R/site-library | ||
|
|
||
| # Install phantompeakqualtools | ||
| ENV SPP_VERSION="1.15" | ||
| ENV PHANTOMPEAKQUALTOOLS_VERSION="v.1.1" | ||
| RUN curl -fsSL https://github.com/hms-dbmi/spp/archive/${SPP_VERSION}.tar.gz -o /opt/SPP_${SPP_VERSION}.tar.gz && \ | ||
| Rscript -e "install.packages('/opt/SPP_${SPP_VERSION}.tar.gz',dependencies=TRUE)" && \ | ||
| rm /opt/SPP_${SPP_VERSION}.tar.gz && \ | ||
| curl -fsSL https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/phantompeakqualtools/ccQualityControl.${PHANTOMPEAKQUALTOOLS_VERSION}.tar.gz -o /opt/phantompeakqualtools.${PHANTOMPEAKQUALTOOLS_VERSION}.tar.gz && \ | ||
| tar xvzf /opt/phantompeakqualtools.${PHANTOMPEAKQUALTOOLS_VERSION}.tar.gz -C /opt/ && \ | ||
| chmod 755 /opt/phantompeakqualtools/* && \ | ||
| echo 'alias run_spp.R="Rscript /opt/phantompeakqualtools/run_spp.R"' >> ~/.bashrc && \ | ||
| rm /opt/phantompeakqualtools.${PHANTOMPEAKQUALTOOLS_VERSION}.tar.gz | ||
| ENV PATH=${PATH}:/opt/phantompeakqualtools | ||
|
|
||
| # Install DeepTools | ||
| ENV DEEPTOOLS_VERSION="2.5.4" | ||
| RUN pip install git+git://github.com/fidelram/deepTools.git@$DEEPTOOLS_VERSION | ||
|
|
||
| # Install ngsplot | ||
| ENV NGSPLOT_VERSION="2.63" | ||
| RUN curl -fsSL https://github.com/shenlab-sinai/ngsplot/archive/${NGSPLOT_VERSION}.tar.gz -o /opt/ngsplot_${NGSPLOT_VERSION}.tar.gz && \ | ||
| tar xvzf /opt/ngsplot_${NGSPLOT_VERSION}.tar.gz -C /opt/ && \ | ||
| rm /opt/ngsplot_${NGSPLOT_VERSION}.tar.gz | ||
| ENV PATH=${PATH}:/opt/ngsplot-${NGSPLOT_VERSION}/bin | ||
| ENV NGSPLOT=/opt/ngsplot-${NGSPLOT_VERSION}/ | ||
| RUN curl -fsSL https://export.uppmax.uu.se/b2013064/test-data/nfcore-chipseq_test_set.tar.bz2 -o /opt/nfcore-chipseq_test_set.tar.bz2 && \ | ||
| tar xvjf /opt/nfcore-chipseq_test_set.tar.bz2 -C /opt/ && \ | ||
| echo y | ngsplotdb.py install /opt/nfcore-chipseq_test_set/ngsplotdb_mm10_75_3.00.tar.gz && \ | ||
| echo y | ngsplotdb.py install /opt/nfcore-chipseq_test_set/ngsplotdb_hg19_75_3.00.tar.gz && \ | ||
| rm /opt/nfcore-chipseq_test_set.tar.bz2 && \ | ||
| rm -rf /opt/nfcore-chipseq_test_set | ||
|
|
||
| # Install MACS | ||
| RUN pip install MACS2 | ||
|
|
||
| # Install MultiQC | ||
| ENV MULTIQC_VERSION="v1.4" | ||
| RUN pip install git+git://github.com/ewels/MultiQC.git@$MULTIQC_VERSION |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| # You can use this file to create a conda environment for this pipeline: | ||
| # conda env create -f environment.yml | ||
| name: nfcore-chipseq-1.4dev | ||
| channels: | ||
| - bioconda | ||
| - conda-forge | ||
| - defaults | ||
| dependencies: | ||
| - conda-forge::openjdk=8.0.144 # Needed for FastQC - conda build hangs without this | ||
| - fastqc=0.11.7 | ||
| - trim-galore=0.4.5 | ||
| - bwa=0.7.17 | ||
| - conda-forge::gawk=4.2.1 | ||
| - conda-forge::r-markdown=0.8 | ||
| - bioconda::r-spp=1.15.2 | ||
| - bedtools=2.27.1 | ||
| - samtools=1.7 | ||
| - picard=2.18.2 | ||
| - deeptools=3.0.2 | ||
| - macs2=2.1.1 | ||
| - multiqc=1.5 | ||
| - conda-forge::r-catools=1.17.1 | ||
| - conda-forge::r-snow=0.4 | ||
| - conda-forge::r-r.utils=2.6.0 | ||
| - conda-forge::r-stringr=1.3.0 | ||
| - conda-forge::r-markdown=0.8 | ||
| - conda-forge::r-evaluate=0.10.1 | ||
| - conda-forge::r-domc=1.3.5 | ||
| - conda-forge::r-ggplot2=2.2.1 | ||
| - conda-forge::r-knitr=1.20 | ||
| - conda-forge::r-rmysql=0.10.13 | ||
| - bioconda::bioconductor-bsgenome=1.46.0 | ||
| - bioconda::bioconductor-rsamtools=1.30.0 | ||
| - bioconda::bioconductor-shortread=1.36.0 | ||
| - bioconda::bioconductor-genomicranges=1.30.3 | ||
| - bioconda::bioconductor-genomicfeatures=1.30.3 | ||
| - bioconda::bioconductor-ensembldb=2.2.2 | ||
| - bioconda::bioconductor-chippeakanno=3.12.0 | ||
| - bioconda::bioconductor-biomart=2.34.2 | ||
| - bioconda::bioconductor-rtracklayer=1.38.3 | ||
| - bioconda::bioconductor-bsgenome.hsapiens.ucsc.hg19=1.4.0 | ||
| - bioconda::bioconductor-org.hs.eg.db=3.5.0 | ||
| - bioconda::bioconductor-bsgenome.mmusculus.ucsc.mm10=1.4.0 | ||
| - bioconda::bioconductor-org.mm.eg.db=3.5.0 | ||
| - deeptools=3.0.2 | ||
| - bioconda::r-spp=1.15.2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had sort of tried to avoid this in the past, as the base docker image already updates. However, I guess we don't know how stale the base container is, so it doesn't hurt 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added that because there was a warn to update it every time asking to update conda. It is not really necessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, the more I think about it, the more I think that it's probably a good idea to have in all nf-core containers..