Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improvements from running complex rnaseq and chipseq #327

Merged
merged 26 commits into from
Apr 4, 2021
Merged

improvements from running complex rnaseq and chipseq #327

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
579c95d
properly specify config overrides in tests
daler Jan 2, 2021
07f9583
better error message for chipseq configs, fixes #243
daler Jan 2, 2021
4d3d15c
use ConfigurationError exception class
daler Jan 2, 2021
bc24eeb
preflight() only cares that first column contains ids
daler Jan 2, 2021
537d281
back to checking if --configfile provided
daler Jan 2, 2021
2e37deb
add config and sampletable for complex chipseq
daler Jan 2, 2021
2d466b0
add driver scripts for complex rnaseq and chipseq
daler Jan 2, 2021
452994e
localrules symlinks only if symlinks rule is created
daler Jan 2, 2021
21821e1
send fastq-dump output to log (rnaseq and chipseq)
daler Jan 2, 2021
c250eaf
use multiple cores for cutadapt in SE mode
daler Jan 2, 2021
953006a
use multiple threads for fastqc
daler Jan 2, 2021
bcf77f6
make sure multibigwigsummary uses configured threads
daler Jan 3, 2021
0b2ed87
support fingerprint in cases where input is unavailable for some samples
daler Jan 3, 2021
b253224
add peak-calling and bigwig merging to complex chipseq config
daler Jan 4, 2021
28356ba
redirect to logs for merge bigwig output
daler Jan 4, 2021
25b8d97
Merge branch 'master' into complex-rnaseq
daler Apr 3, 2021
809c3bd
Merge branch 'mamba-for-v1.7rc' into complex-rnaseq
daler Apr 4, 2021
fc87b84
fix configfile overrides in tests
daler Apr 4, 2021
495ad75
update env-pinned.yml
daler Apr 4, 2021
99851a1
use pinned env for deploy and tests
daler Apr 4, 2021
b979450
update env-r-pinned
daler Apr 4, 2021
9dbb27f
update cache key to reflect env yamls
daler Apr 4, 2021
8cb5813
Merge branch 'v1.8rc' into complex-rnaseq
daler Apr 4, 2021
7f590a2
add additional test configs
daler Apr 4, 2021
30b22d7
when testing urls, skip file://
daler Apr 4, 2021
3f96762
fix name of test rnaseq config
daler Apr 4, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 17 additions & 14 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ variables:

save_cache: &save_cache
save_cache:
key: v0-{{ checksum "env.yml" }}-{{ checksum "env-r.yml" }}
key: v0-{{ checksum "env-pinned.yml" }}-{{ checksum "env-r-pinned.yml" }}-{{ checksum "env-r.yml" }}- {{ checksum "env.yml" }}
paths:
- /opt/conda

Expand All @@ -38,7 +38,7 @@ variables:
restore_cache: &restore_cache
restore_cache:
keys:
- v0-{{ checksum "env.yml" }}-{{ checksum "env-r.yml" }}
- v0-{{ checksum "env-pinned.yml" }}-{{ checksum "env-r-pinned.yml" }}-{{ checksum "env-r.yml" }}- {{ checksum "env.yml" }}

# --------------------------------------------------------------------------
# The path needs to be set each time; in jobs below this will be called as
Expand Down Expand Up @@ -71,11 +71,9 @@ variables:
conda config --system --add channels bioconda
conda config --system --add channels conda-forge
conda install mamba -y
mamba env create -n lcdb-wf-test --file env.yml
mamba env create -n lcdb-wf-test-r --file env-r.yml
mamba env create -n lcdb-wf-test --file env-pinned.yml
mamba env create -n lcdb-wf-test-r --file env-r-pinned.yml
fi
# conda env export -n lcdb-wf-test > env.yaml
# conda env export -n lcdb-wf-test-r > env-r.yaml

# --------------------------------------------------------------------------
# Deploy into a new directory and get the test data.
Expand Down Expand Up @@ -154,15 +152,15 @@ variables:
./run_test.sh --use-conda -j2 -k -p -r \
--configfile $ORIG/test/test_configs/test_chipseq_regression.yaml \
--config sampletable=$ORIG/test/test_configs/chipseq_one_run.tsv \
--config merged_bigwigs="{}" \
merged_bigwigs="{}" \
--until bed_to_bigbed

# Piggy-backing on that test, here we check to make sure it's OK to
# omit peak-calling config. We just do a dry-run.
./run_test.sh -n \
--configfile $ORIG/test/test_configs/test_chipseq_no_peaks.yaml \
--config sampletable=$ORIG/test/test_configs/chipseq_one_run.tsv \
--config merged_bigwigs="{}" \
merged_bigwigs="{}" \
--until bed_to_bigbed

# --------------------------------------------------------------------------
Expand Down Expand Up @@ -218,19 +216,19 @@ variables:
# only go to the cutadapt step.
rm -r data
./run_test.sh -j 1 --use-conda -k -p -r --until cutadapt \
--configfile $ORIG/test/test_configs/override.yaml \
--configfile $ORIG/test/test_configs/test_rnaseq_config.yaml \
--config sampletable=$ORIG/test/test_configs/test_sra_sampletable.tsv

# Same as above, but SE
rm -r data
./run_test.sh -j 1 --use-conda -k -p -r --until cutadapt \
--configfile $ORIG/test/test_configs/override.yaml \
--configfile $ORIG/test/test_configs/test_rnaseq_config.yaml \
--config sampletable=$ORIG/test/test_configs/test_sra_sampletable_SE_only.tsv

# test strandedness, PE
rm -r data && cp -r /tmp/data data
./run_test.sh -j 2 --use-conda -k -p -r --until strand_check \
--configfile $ORIG/test/test_configs/override.yaml \
--configfile $ORIG/test/test_configs/test_rnaseq_config.yaml \
--config sampletable=$ORIG/test/test_configs/test_pe_sampletable.tsv \
&& cat strandedness.tsv

Expand All @@ -245,22 +243,27 @@ variables:
rm -r data && cp -r /tmp/data data
./run_test.sh -j 2 --use-conda -k -p -r \
--forcerun star_pass1 \
--configfile $ORIG/test/test_configs/star_override_2pass.yaml \
--configfile \
$ORIG/test/test_configs/test_rnaseq_config.yaml \
$ORIG/test/test_configs/star_override_2pass.yaml \
--config sampletable=$ORIG/test/test_configs/two_samples.tsv \
--until star_pass2

# test STAR 1-pass alignment
rm -r data && cp -r /tmp/data data
./run_test.sh -j 2 --use-conda -k -p -r \
--forcerun star \
--configfile $ORIG/test/test_configs/star_override_1pass.yaml \
--configfile \
$ORIG/test/test_configs/test_rnaseq_config.yaml \
$ORIG/test/test_configs/star_override_1pass.yaml \
--config sampletable=$ORIG/test/test_configs/two_samples.tsv \
--until star

# test PE reads
rm -r data && cp -r /tmp/data data
./run_test.sh -j 2 --use-conda -k -p -r --until multiqc \
--configfile $ORIG/test/test_configs/override.yaml \
--configfile \
$ORIG/test/test_configs/test_rnaseq_config.yaml \
--config sampletable=$ORIG/test/test_configs/test_pe_sampletable.tsv

# --------------------------------------------------------------------------
Expand Down
4 changes: 2 additions & 2 deletions deploy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ def filter_out_other_workflows(filenames, patterns_to_include, prefilter="workfl
"-p",
"./env",
"--file",
"env.yml",
"env-pinned.yml",
"-c",
"conda-forge",
"-c",
Expand All @@ -242,7 +242,7 @@ def filter_out_other_workflows(filenames, patterns_to_include, prefilter="workfl
"-p",
"./env-r",
"--file",
"env-r.yml",
"env-r-pinned.yml",
"-c",
"conda-forge",
"-c",
Expand Down
Loading