4.6.0.0

[droazen]

Download release: gatk-4.6.0.0.zip
Docker image: https://hub.docker.com/r/broadinstitute/gatk/

Highlights of the 4.6.0.0 release:

• We've fixed a serious CRAM writing bug that affects GATK versions 4.3 through 4.5 and Picard versions 2.27.3 through 3.1.1. This bug can, in limited cases, lead to reads with an incorrect base sequence being written. See this comment to GATK issue 8768 and the full release notes below for more details on what conditions trigger the bug.

  • To help users detect whether their CRAM files are affected, we've released a CRAM scanning tool called CRAMIssue8768Detector that can detect whether a particular CRAM file is affected by this bug. If you suspect that some of your CRAM files may have been affected, please run this tool on them for confirmation!

• By overwhelming popular demand, we've switched back to using the standard ./. representation for no-calls in GenotypeGVCFs and GenomicsDB instead of 0/0 with DP=0. This reverts the change described in our article GenotypeGVCFs and the death of the dot.

  • We intend to publish a new article shortly to replace that older article with further details on this change. When we do so, we'll link to it from here.

• The Mutect2 germline resource can now have split multiallelic format

• Added an --inverted-read-filter argument to allow for selecting reads that fail read filters from the command line easily

• We've fixed a number of issues with HTTP support, mainly affecting the loading of side inputs such as indices over HTTP

• Reduced the number of layers in the GATK docker image to help users running into docker quota issues

Full list of changes:

• Important CRAM writing bug fix and detection tool

  • We've updated to HTSJDK 4.1.1 and Picard 3.2.0 (Update HTSJDK to 4.1.1 and Picard to 3.2.0 #8900), which fix a serious bug in the CRAM writing code first reported in GATK issue 8768
  • This issue affects GATK versions 4.3.0.0 through 4.5.0.0, and is fixed in GATK 4.6.0.0.
  • This issue also affects Picard versions 2.27.3 through 3.1.1, and is fixed in Picard 3.2.0.
  • The bug is triggered when writing a CRAM file using one of the affected GATK/Picard versions, and both of the following conditions are met:
    • At least one read is mapped to the very first base of a reference contig
    • The file contains more than one CRAM container (10,000 reads) with reads mapped to that same reference contig
  • When both of these conditions are met, the resulting CRAM file may have corrupt containers associated with that contig containing reads with an incorrect sequence.
  • Since many common references such as hg38 have N's at the very beginning of the autosomes and X/Y, many pipelines will not be affected by this bug. However, users of a telomere-to-telomere reference, users doing mitochondrial calling, and users with reads aligned to the alt sequences will want to scan their CRAM files for possible corruption.
  • The other mitigating circumstance is that when a CRAM is affected, the signal will be overwhelmingly obvious, with the mismatch rate typically jumping from sub-1% to 80-90% for the affected regions, making it likely to be caught by standard QC processes.
  • We've released a CRAM scanning tool called CRAMIssue8768Detector (Tool to detect CRAM base corruption caused by GATK issue 8768 #8819) that can detect whether a particular CRAM file is affected by this bug. If you suspect that some of your CRAM files may have been affected, please run this tool on them for confirmation!

• Joint Calling

  • We've switched back to using the standard ./. representation for no-calls in GenotypeGVCFs and GenomicsDB instead of 0/0 with DP=0 ( fix no data hom refs #8715) (GGVCFs changes to revert "no data" calls to "./." and change GQ0 hom-refs to no-calls #8741) (Allow for GT to be a nocall if GQ and PL[0] are zero instead of homref in GenomicsDB #8759)
    • This reverts the change described in our article GenotypeGVCFs and the death of the dot
  • Fix for GenotypeGVCFs with mixed ploidy sites (Fix GenotypeGVCFs with mixed ploidy sites #8862)
  • Fix for GnarlyGenotyper when PLs are null (Fix for gnarly when PLs are null #8878)
  • Fixed bug in ReblockGVCF when removing annotations (Fixing bug when removing annotations from empty map #8870)
  • Enable ReblockGVCF to subset AS annotations that aren't "raw" (pipe-delimited) (Enable ReblockGVCF to subset AS annotations that aren't "raw" (pipe-delimited) #8771)
  • Remove header lines in ReblockGVCF when we remove FORMAT annotations (Remove header lines in ReblockGVCFs when we remove FORMAT annotations #8895)
  • ReblockGVCF: Add malaria spanning deletion exception regression test with fix (Add malaria spanning deletion exception regression test with fix #8802)
  • Restore some GnarlyGenotyper tests (Restore gnarly tests #8893)

• HaplotypeCaller

  • Fix to long deletions that overhang into the assembly window causing exceptions in HaplotypeCaller (Fix  #8731)

• Mutect2

  • The Mutect2 germline resource can now have split multiallelic format (Mutect2 germline resource can have split multiallelic format #8837)
  • Make the Mutect2 haplotype and clustered events filters smarter about germline events (Make M2 haplotype and clustered events filters smarter about germline events #8717)
  • Added the DragSTR model to the Mutect2 WDL (dragstr model in Mutect2 WDL #8716)
  • Improvements to Mutect2's Permutect training data mode (Improvements to Mutect2's Permutect training data mode #8663)
  • Bigger Permutect tensors and Permutect test datasets can be annotated with truth VCF (Bigger Permutect tensors and Permutect test datasets can be annotated with truth VCF #8836)
  • Mutect2 WDL and GetSampleName can handle multiple sample names in BAM headers (Mutect2 WDL and GetSampleName can handle multiple sample names in BAM headers #8859)
  • Permutect dataset engine outputs contig and read group indices, not names (Permutect dataset engine outputs contig and read group indices, not names #8860)
  • Normal artifact LOD is now defined without the extra minus sign (normal artifact lod is now defined without the extra minus sign #8668)

• CNV Calling

  • Fixed the GT header in PostprocessGermlineCNVCalls's --output-genotyped-intervals output (Fix GT header in PostprocessGermlineCNVCalls's --output-genotyped-intervals output #8621)

• SV Calling

  • Reduced SVConcordance memory footprint (Reduce SVConcordance memory footprint #8623)
  • Rewrote complex SV functional annotation in SVAnnotate (Rewrite complex SV functional annotation in SVAnnotate #8516)
  • We now handle the CTX_INV subtype in SVAnnotate (Handle CTX_INV subtype in SVAnnotate #8693)

• Flow-based Calling

  • SNVQ recalibration tool added for flow-based reads (SNVQ recalibration for flow based reads #8697)
  • Bug fix in flow-based allele filtering (Bug fix in flow allele filtering  #8775)
  • Fixed a bug in flow-based AlleleFiltering that ignored more than a single sample (Fixed a bug in AlleleFiltering that ignored more than a single sample #8841)
  • Fixed an edge case in flow-based variant annotation ([BIOIN-1570] Fixed edge case in variant annotation  #8810)

• Notable Enhancements

  • Added an --inverted-read-filter argument to allow for selecting reads that fail read filters from the command line easily (added --inverted-read-filter argument to allow for selecting reads failing read filters from the command line easily #8724)
  • Inverted SoftClippedReadFilter to conform to the standard filtering logic (Inverted SoftClippedReadFilter to conform to filtering logic #8888)
  • Reduced the number of docker layers in the GATK image from 44 to 16 (GATK Reduced Docker Layers for ACR #8808)
  • VariantFiltration: added a --mask-description argument to write custom mask filter description in VCF header (Adding arg to VariantFiltration to write your own description in Header #8831)
  • GatherVcfsCloud is no longer beta (GatherVcfsCloud is no longer Beta #8680)

• Miscellaneous Changes

  • GetPileupSummaries now uses the standard MappingQualityReadFilter instead of a custom --min-mapping-quality argument (Gc getpipeupsummaries use mappingqualityreadfilter #8781)
  • Funcotator: suppress a log message about b37 contigs when not doing b37/hg19 conversion (Funcotator: suppress a log message about b37 contigs when not doing b37/hg19 conversion #8758)
  • Output the new image name at the end of a successful cloud docker build (Output the new image name at the end of a successful cloud docker build #8627)
  • Exclude the test folder from code coverage calculations (Exclude test folder from codecov scanning #8744)
  • Removed deprecated genomes in the cloud docker image that was causing CNN WDL test failures (removing deprecated genomes in the cloud docker image that was causing test failures #8891)
  • Re-commit large test files as lfs stubs (Change large test files from PR 8741 to lfs #8769)
  • Standardize test results directory between normal/docker tests (Standardize test results directory between normal/docker tests #8718)
  • Improve failure message in VariantContextTestUtils (Improve failure message in VariantContextTestUtils #8725)
  • Update the setup_cloud github action (Update the setup_cloud github action #8651)
  • Parameterize the logging frequency for ProgressLogger in GatherVcfsCloud (Parameterize the logging frequency for ProgressLogger. #8662)

• Documentation

  • Updated the README to include list of popular software included in docker image (Update README to include list of popular software included in docker image #8745)

• Dependencies

  • Updated HTSJDK to 4.1.1, which fixes the CRAM writing bug described above (Update HTSJDK to 4.1.1 and Picard to 3.2.0 #8900)
  • Updated Picard to 3.2.0, which fixes the CRAM writing bug described above (Update HTSJDK to 4.1.1 and Picard to 3.2.0 #8900)
  • Updated GenomicsDB to 1.5.3, which supports M1 Macs and switches no-call representation back to ./. (Move to GenomicsDB 1.5.2 #8710) (Allow for GT to be a nocall if GQ and PL[0] are zero instead of homref in GenomicsDB #8759)
  • Updated http-nio to 1.1.1, which fixes several URL-handling bugs with HTTP support (Update http-nio to 1.1.1 #8889)
  • Updated several miscellaneous dependencies to fix security vulnerabilities (Update several dependencies to fix vulnerabilities #8898)

---

This discussion was created from the release 4.6.0.0.

[Scott-0208]

How do I download it, install it, and use it in the Ubuntu terminal? Thank you. I have already downloaded it and extracted it. But I don't know how do I use it.

[droazen]

@Scott-0208 From the directory where you extracted the GATK zip file, run ./gatk --help, which will tell you how to list the available tools, and get help on a specific tool. You can also read the tutorials and documentation on our website (https://gatk.broadinstitute.org/hc/en-us), or ask a question on our user forums (https://gatk.broadinstitute.org/hc/en-us/community/topics).

[jin0008]

When doing GenotypeGVCFs, significant memory issue occurred. The process was interrupted when I was doing genotypeGVCFs from GenomicsDB on 420 samples. 4.5.0.0 version is fine. Could you please fix this?

[Ben-Habermeyer]

@droazen will 4.6.0.0 be available from bioconda https://anaconda.org/bioconda/gatk4 ? Thank you!

[droazen]

@Ben-Habermeyer We don't maintain the GATK bioconda distribution, but I was told by bioconda maintainer @matthdsm in #8561 (comment) that they are unable to "get the latest version of the GATK to build because of the requirements for the gcnvkernel", and are waiting for us to release the Python changes that were merged in that PR. We are planning another GATK release to get those Python changes out in the early-August timeframe.

[Ben-Habermeyer]

Thank you for the quick reply - I will look out for that release!

[Ben-Habermeyer]

Hi @droazen just following up on this discussion to see if there are any changes to the proposed release timeframe. Thanks again

[droazen]

@Ben-Habermeyer Yes, the next release has been held up a bit longer than anticipated due to complications arising from the Python environment changes -- we are now targeting a release this month (September).