Benchmarking

[sorelfitzgibbon]

I'm collecting various benchmarking threads here.

1. mosdepth and CollectWgsMetrics

[sorelfitzgibbon]

@tyamaguchi-ucla @raagagrawal @yashpatel6 @pboutros These results have now been updated to include Raag's high coverage sample. I think we should drop per-base-coverage output. Let me know if you want plots.

mosdepth vs SAMtools depth

The results below show that the two tools can be made to give almost identical per base results by setting min-BQ 0. Only SAMtools depth has the option to change min-BQ, and using this does have a noticeable impact on per base coverages.
Per base coverages are used for downstream statistical calculations requiring knowledge of how many sites could possibly have had a variant. The differences when setting SAMtools depth to min-BQ 20 remain small overall and could be acceptable, however I suggest we should only output coverage windows with this pipeline and we should instead use the gVCFs from the call-gSNP pipeline for the non-variant reference quality information, for these reasons:

• the g.vcfs are already being created by the pipeline
  • avoid creating extra ~8 GB per-base output file
• the reference confidence scores within the g.vcfs are based on more than just base quality and they are calculated after local reassembly by HaplotypeCaller or Mutect2, thus presumably giving more accurate results.
• mosdepth is much less resource intense: 3G vs 26G and parallelizable.

Samples

Resources

Per base coverage differences

• parameters used
  • mosdepth --mapq 20
  • samtools depth -s -min-MQ 20 -min-BQ $x
  • all other parameter options were default
  • more parameter option details here

Results for HG001 (30x), SARC (30x) and BAPR (80x) and min-BQ 0, 5, 10, 20.

Cause of non-base-quality differences

For the HG001 sample, of the 3,532 positions with coverage greater than 5x, only 379 have coverage less than 100x indicating the differences occur mostly at very high coverage sites (3/4 are > 1000x)
- the normal coverage sites have minor/rare issues, for example:

For the positions shown below with the 2 base deletion, mosdepth calls 26x and samtools calls 20x. There are 6 shaded deletions which represent read pairs where one read has the deletion and the other doesn't. (This is caused by one of the reads ending by the simple repeat region and aligning with a single mismatch instead of the 2 base deletion). `mosdepth