Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

host_filter.wdl modernization #70

Merged
merged 83 commits into from
Apr 25, 2023
Merged
Changes from 1 commit
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
e92a5e0
fastp
mlin May 20, 2022
24f6841
fastp single
mlin May 25, 2022
06517ea
bowtie2 run
mlin May 26, 2022
6c485e4
hisat2 run
mlin May 26, 2022
623a906
dedup run
mlin May 26, 2022
0c75c04
run subsample
mlin May 26, 2022
c45b5c3
run kallisto
mlin May 27, 2022
18917cc
adjust index tar filenames
mlin May 29, 2022
029ab72
polishing
mlin May 30, 2022
15d65fe
polishing
mlin May 30, 2022
bc8aebe
count reads in each step
mlin May 31, 2022
db223c8
Create host_filter_indexing.wdl
mlin May 31, 2022
d0a5ebe
boost fastp complexity threshold
mlin Jun 14, 2022
b60afa1
output fastp report
mlin Jun 14, 2022
8e308bd
build fastp from our fork with SDUST complexity filtering
mlin Jul 31, 2022
17e31c5
use fastp --sdust_complexity_filter
mlin Jul 31, 2022
7bcb00f
bump
mlin Aug 1, 2022
b70ce8f
bump
mlin Aug 1, 2022
052120e
tune
mlin Aug 1, 2022
169336f
stub the remaining step descriptions
mlin Aug 3, 2022
f2e27ca
wire to tests
mlin Aug 10, 2022
d7b3959
and auto_benchmark
mlin Aug 10, 2022
22cc6c2
fixup tests
mlin Aug 11, 2022
0089204
fixup tests
mlin Aug 11, 2022
a17a508
fixup tests
mlin Aug 12, 2022
f96344e
fixup tests
mlin Aug 12, 2022
97fbb83
fixup tests
mlin Aug 12, 2022
101c8dc
fixup tests
mlin Aug 12, 2022
4b2fbea
add back in picard CollectInsertSizeMetrics
mlin Sep 5, 2022
5317bb8
picard step description
mlin Sep 7, 2022
6ef7035
host_filter_2022.wdl => host_filter.wdl
mlin Sep 7, 2022
5062f19
polish
mlin Sep 8, 2022
713d414
restore fastqs_0 and fastqs_1 to minimize collateral changes
mlin Sep 8, 2022
9e2612e
add minimap2 index build
mlin Sep 22, 2022
ca47e36
picard_insert_metrics.txt
mlin Sep 22, 2022
378573c
amr/run.wdl workaround
mlin Sep 22, 2022
aeeb7bc
index multiple transcripts_fasta_gz
mlin Oct 23, 2022
a897fe6
make gtf optional
mlin Oct 23, 2022
0d66e58
allow uncompressed genome fasta
mlin Oct 30, 2022
1606cc9
allow uncompressed genome fasta
mlin Oct 30, 2022
0d0cd69
allow uncompressed genome fasta
mlin Oct 30, 2022
de5b5c1
bump minimap2 memory
mlin Oct 30, 2022
9307f80
bump minimap2 memory
mlin Oct 30, 2022
03d367e
step descriptions -- first draft
mlin Nov 2, 2022
10d3219
add indexing driver & draft readme
Nov 2, 2022
7126058
include invocations in step descriptions
mlin Nov 4, 2022
afcb3fb
rebase amr fix
mlin Nov 16, 2022
9fb0824
load card_json
valenzuelaomar Jan 26, 2023
0071184
run kallisto every time
valenzuelaomar Jan 26, 2023
06db702
fix amr wdl
valenzuelaomar Jan 27, 2023
14ba43b
fix short-read-mngs rebase weirdness
valenzuelaomar Jan 27, 2023
884b8a2
add final things
valenzuelaomar Jan 27, 2023
ff8b815
[modernized host filter] add ERCC and gene-level outputs to kallisto …
mlin Jan 30, 2023
c29ed76
load card_json explicitly
valenzuelaomar Jan 30, 2023
448f997
add ~
valenzuelaomar Jan 30, 2023
61d8767
fix host_filter unit tests
mlin Feb 1, 2023
9e1ab88
fix host_filter unit tests
mlin Feb 1, 2023
f9c49ac
bowtie2: sort by read name for better reproducibility
mlin Feb 1, 2023
33421ed
update minimap2 indexing invocation
mlin Feb 2, 2023
7cc1bbe
add chelonia_mydas, drosophila_melanogaster, gray_whale, pea-aphid
mlin Feb 3, 2023
fa870a6
copy-paste {bowtie2,hisat2}_human_filter to support pipeline viz
mlin Feb 4, 2023
110574c
allow kallisto nonzero exit
mlin Feb 5, 2023
cea0ba7
rename modern host filtering inputs/outputs and create a 1-1 mapping …
valenzuelaomar Feb 9, 2023
20fd60d
fix lint issue
valenzuelaomar Feb 9, 2023
a1ae1c2
Merge remote-tracking branch 'origin/main' into mlin/modernize-host-f…
mlin Feb 9, 2023
2a41734
rename reads_in_count to input_read_count
mlin Feb 9, 2023
96f77b7
auto_benchmark updates
mlin Feb 9, 2023
bf3a134
fix test_RunCZIDDedup_safe_csv
mlin Feb 9, 2023
1aba733
rename kallisto output files
mlin Feb 9, 2023
35636e4
update mosquitos with several Culicidae
mlin Feb 12, 2023
90cea20
add files to wdl output for pipeline viz compatibility
valenzuelaomar Feb 16, 2023
81ea0ad
convert headers in descriptions to bolded text
valenzuelaomar Feb 16, 2023
40d2052
Merge branch 'main' into mlin/modernize-host-filter
mlin Mar 31, 2023
e35cd04
delete host_filter_indexing since it's subsumed in #182
mlin Mar 31, 2023
8e981c4
Merge branch 'main' into mlin/modernize-host-filter
valenzuelaomar Apr 14, 2023
aeb234f
fix glob patterns in read counting
mlin Apr 18, 2023
74304e0
Revert "fix glob patterns in read counting"
mlin Apr 18, 2023
ec91bf3
[Bug] fix count expansion for single file short-read-mngs (#216)
rzlim08 Apr 18, 2023
021d183
Merge branch 'main' into mlin/modernize-host-filter
valenzuelaomar Apr 19, 2023
227a489
Remove AMR changes that are a WIP from modern host filtering branch (…
valenzuelaomar Apr 19, 2023
3ad5ad0
tune hisat2 memory usage (#223)
rzlim08 Apr 21, 2023
3976349
Legacy Host Filter initial commit (#224)
rzlim08 Apr 25, 2023
ba46f53
Revert "Remove AMR changes that are a WIP from modern host filtering …
rzlim08 Apr 25, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fixup tests
mlin authored and valenzuelaomar committed Jan 27, 2023
commit a17a5085bcab19370a52207d791b8fd3268479cb
2 changes: 1 addition & 1 deletion workflows/short-read-mngs/auto_benchmark/harvest.py
Original file line number Diff line number Diff line change
@@ -93,7 +93,7 @@ def harvest_sample(sample, outputs_json, taxadb):
outputs_json["czid_short_read_mngs.host_filter.fastp_out_fastp2_fastq"]
is not None
)
ans["input_reads"] = read_output_jsonfile(outputs_json, "host_filter.input_read_count")[
ans["input_reads"] = read_output_jsonfile(outputs_json, "host_filter.reads_in_count")[
"fastqs"
]
for step in [
Original file line number Diff line number Diff line change
@@ -28,7 +28,7 @@ def test_RunCZIDDedup_safe_csv(util, short_read_mngs_bench3_viral_outputs):
input_file.seek(0)
assert special_char_rows == quote_count

inputs["reads1_fastq"] = [input_file.name]
inputs["reads1_fastq"] = input_file.name

outp = util.miniwdl_run(
util.repo_dir() / "workflows/short-read-mngs/host_filter.wdl",