4.4.0.0

[droazen]

Download release: gatk-4.4.0.0.zip
Docker image: https://hub.docker.com/r/broadinstitute/gatk/

Highlights of the 4.4.0.0 release:

• We've moved to Java 17, the latest long-term support (LTS) Java release, for building and running GATK! Previously we required Java 8, which is now end-of-life.

  • Newer non-LTS Java releases such as Java 18 or Java 19 may work as well, but since they are untested by us we only officially support running with Java 17.

• Significant enhancements to SelectVariants, including arguments to enable GVCF filtering support and to work with genotype fields more easily.

• A new tool SVConcordance, that calculates SV genotype concordance between an "evaluation" VCF and a "truth" VCF

• Bug fixes and enhancements to the support for the Ultima Genomics flow-based sequencing platform introduced in GATK 4.3.0.0

Full list of changes:

• Flow-based Variant Calling

  • FlowFeatureMapper: added surrounding-median-quality-size feature (FlowFeatureMapper: added surrounding-median-quality-size feature #8222)
  • Removed hardcoded limit on max homopolymer call (Remove hardcoded limit on max homopolymer call #8088)
  • Fixed bug in dynamic read disqualification (bug in dynamic read disqualification fixed #8171)
  • Fixed a bug in the parsing of the T0 tag (Fixing a bug in parsing of T0 tag #8185)
  • Updated flow-based calling Mutect2 parameters to make them consistent with the HaplotypeCaller parameters (Updated M2 parameters to make them consistent with HC #8186)

• SelectVariants

  • Enabled GVCF type filtering support in SelectVariants (Enabled GVCF type filtering support in SelectVariants #7193)
    • Added an optional argument --ignore-non-ref-in-types to support correct handling of VariantContexts that contain a NON_REF allele. This is necessary because every variant in a GVCF file would otherwise be assigned the type MIXED, which makes it impossible to filter for e.g. SNPs.
    • Note that this only enables correct handling of GVCF input. The filtered output files are VCF (not GVCF) files, since reference blocks are not extended when a variant is filtered out.
  • SelectVariants: added new arguments for controlling genotype JEXL filtering (SelectVariants JEXL filter fixes and refactor #8092)
    • -select-genotype: with this new genotype-specific JEXL argument, we support easily filtering by genotype fields with expressions like 'GQ > 0', where the behavior in the multi-sample case is 'GQ > 0' in at least one sample. It's still possible to manually access genotype fields using the old -select argument and expressions such as vc.getGenotype('NA12878').getGQ() > 0.
    • --apply-jexl-filters-first: This flag is provided to allow the user to do JEXL filtering before subsetting the format fields, in particular the case where the filtering is done on INFO fields only, which may improve speed when working with a large cohort VCF that contains genotypes for thousands of samples.

• SV Calling

  • Added a new tool SVConcordance, that calculates SV genotype concordance between an "evaluation" VCF and a "truth" VCF (Implement SVConcordance tool #7977)
  • Recognize MEI DELs with ALT format DEL:ME in SVAnnotate (Recognize MEI DELs with ALT format <DEL:ME> in SVAnnotate #8125)
  • Don't sort rejected reads output from AnalyzeSaturationMutagenesis (don't sort output #8053)

• Notable Enhancements

  • GenotypeGVCFs: added an --keep-specific-combined-raw-annotation argument to keep specified raw annotations (Adding argument to GenotypeGVCFs to keep only RAW_GT_COUNT #7996)
  • VariantAnnotator now warns instead of fails when the variant contains too many alleles (VariantAnnotator fails when the variant contains too many alleles #8075)
  • Read filters now output total reads processed in addition to the number of reads filtered (Added a slightly more helpful error message to the reads filter #7947)
  • Added GenomicsDB arguments to the CreateSomaticPanelOfNormals tool (Expose GenomicsDBArgumentCollection with CreateSomaticPanelOfNormals #6746)
  • Added a DeprecatedFeature annotation and a process for officially marking GATK tools as deprecated (Add DeprecatedFeature annotation. #8100)
  • Prevent tool close() methods from hiding underlying errors (Prevent close from hiding causal exceptions #528 #7764)

• Bug Fixes

  • Fixed issue causing VariantRecalibrator to sometimes fail if user provided duplicate -an options (Fixed issue causing VariantRecalibrator to sometimes fail if user provides duplicate -an options #8227)
  • ReblockGVCF: remove A,R, and G length attributes when ReblockGVCF subsets an allele (removing A,R, and G length attributes when ReblockGVCF subsets an allele #8209)
    • Previously if an input gVCF had allele length, reference length, or genotype length annotations in the FORMAT field, ReblockGVCF would not remove all of them at sites where an allele was dropped. This makes the output gVCF invalid since the annotation length no longer matches the length described in the header at those sites. Now we fix up F1R2, F2R1, and AF annotations and remove any other annotations that are not already handled that are defined as A, R, or G length in the header.
  • Fixed a gCNV bug that breaks the inference when only 2 intervals are provided (Number of intervals edge case gCNV fix #8180)
  • Fixed NPE from unintialized logger in GenotypingEngine (Fix NPE from unintialized logger in GenotypingEngine #8159)
  • Fixed asynchronous Python exception propagation in StreamingPythonExecutor/CNNScoreVariants (Fix asynchronous Python exception propagation in StreamingPythonExecutor/CNNScoreVariants. #7402)
  • Fixed issue in ShiftFasta where the interval list output was never written (Fixing issue in ShiftFasta where the intervals aren't written #8070)
  • Bugfix for the type of some output files in the somatic CNV WDL (bugfix for type of some output files in cnv_somatic_pair_workflow.wdl #6735) (Removed commented lines from somatic CNV WDL. #8130)
  • MergeAnnotatedRegions now requires a reference as asserted in its documentation (require reference as asserted in documentation #8067)

• Miscellaneous Changes

  • Deprecated an untested VariantRecalibrator argument and an old ReblockGVCF argument that produced invalid GVCFs (Deprecate some args #8140)
  • Removed old GnarlyGenotyper code with a diploid assumption to prepare for adding haploid support to GnarlyGenotyper (Deprecate some args #8140)
  • ReblockGVCF: add error message for when tree-score-threshold is set but the TREE_SCORE annotation is not present (add error message for when tree-score-threshold is set in ReblockGVCF… #8218)
  • TransferReadTags: allow empty unaligned bams as input (Allow empty input in TransferReadTags #8198)
  • Refactored JointVcfFiltering WDL and expanded tests. (Refactored JointVcfFiltering WDL and expanded tests. #8074)
  • Updated the carrot github action workflow to the most recent version, which supports using #carrot_pr to trigger branch vs master comparison runs (Updated carrot publish github action version #8084)
  • Replaced uses of File.createTempFile() with IOUtils.createTempFile() to ensure that temp files are deleted on shutdown (Replace File.createTempFile with IOUtils.createTempFile #6780)
  • Don't require python just to instantiate the CNNScoreVariants tool classes. (Don't require python to just instantiate tool classes. #8128)
  • Made several Funcotator methods and fields protected so it is easier to extend the tool (Make several Funcotator methods and fields protected instead of private #8124) (Make several Funcotator methods and fields protected #8166)
  • Test for presence of ack result message and simplify ProcessControllerAckResult API (Test for presence of ack result message and simplify ProcessControllerAckResult API #7816)
  • Fixed the path reported by the gatkbot when there are test failures (Fixing the path reported by the gatkbot when there are test failures #8069)
  • Fixed incorrect boolean value in DirichletAlleleDepthAndFractionIntegrationTest (Fixed the broken test in DirichletAlleleDepthAndFractionIntegrationTest #7963)
  • Removed two ancient and unused HaplotypeCaller test files that are no longer needed (Removed GATK3.5 VCFs from HC integration test files.  #7634)
  • Added scattered gCNV case WDL to dockstore file (Added scattered gcnv case wdl to dockstore file #8217)

• Documentation

  • Updated instructions for installing Java in the README (Update instructions for installing Java 8 in the README #8089)
  • Added documentation on OMP_NUM_THREADS and MKL_NUM_THREADS to GermlineCNVCaller and DetermineGermlineContigPloidy (Added documentation on OMP_NUM_THREADS and MKL_NUM_THREADS to GermlineCNVCaller and DetermineGermlineContigPloidy. #8223)
  • Improvements to PileupDetectionArgumentCollection documentation (Small improvements to the pileup arguments #8050)
  • Fixed typo in documentation for VariantAnnotator (Fix minor documentation mistake in VariantAnnotator #8145)

• Dependencies

  • Moved to Java 17, the latest LTS Java release, for building/running GATK (Java 17 and Spark 3.3.0. #8035)
  • Updated Gradle to 7.5.1 (Update Gradle to 7.5.1 #8098)
  • Updated the GATK base docker image to 3.0.0 (Update base image #8228)
  • Updated HTSJDK to 3.0.5 (Java 17 and Spark 3.3.0. #8035)
  • Updated Picard to 3.0.0 (Java 17 and Spark 3.3.0. #8035)
  • Updated Barclay to 5.0.0 (Java 17 and Spark 3.3.0. #8035)
  • Updated GenomicsDB to 1.4.4 (Move to GenomicsDB 1.4.4 #7978)
  • Updated Spark to 3.3.1 (Java 17 and Spark 3.3.0. #8035)
  • Updated Hadoop to 3.3.1. (Use Java 11.0.11+9 for testing, upgrade hadoop to v3.3.1. #8102)
  • Require commons-text 1.10.0 to fix a security vulnerability (Add constraint to transitive dependency commons-text #8071)

---

This discussion was created from the release 4.4.0.0.

[jinyanjane]

I met a problem with run CFSAN SNP pipeline. Error: CLASSPATH is not configured with the path to GenomeAnalysisTK.jar.
I have installed the lastest GATK version 4.4.0.0. And in my ~/.bashrc file, the classfath setting is: export CLASSPATH=/root/gatk-4.4.0.0/gatk-package-4.4.0.0-local.jar:$CLASSPATH

I have checked the path and everything is fine in my sense. Can you help me? It is an urgency.

Thank you.

[carbocation]

The docker image for 4.4.0.0 is 2.81 GB, which is 1.2 GB larger (75% larger) than the 4.3.0.0 tag, which was 1.63 GB. (See dockerhub for confirmation.)

Is there a reason why 4.4.0.0 is so much larger than 4.3.0.0?