[backport] DeepTau - Do not call TF inference with empty grid

[valsdav]

PR description:

Fix DeepTau issue #44333.
Backport to 14_0_X of #44455 .

[2] Calling method for module DeepTauId/'hltHpsPFTauDeepTauProducerForVBFIsoTau'
Exception Message:
error while running session: INVALID_ARGUMENT: Incompatible shapes: [0,1,1,64] vs. [154]

The number of valid_grid_cells here is 0 for some events and this is creating a TF::Tensor with shape [0, 1, 1, N].

When this input is passed to a TF model executed on a CPU without AVX512F AVX512_VNNI, the model is executed and returns an empty output without complaining. When AVX512F AVX512_VNNI instructions are present, the TF executor complains.

This PR avoids calling the TF inference if empty inputs are detected.

The grid should never be empty. The issue will be followed up with Tau experts (enter here issue number..)

PR validation:

Works on the issue reproducer from #44333 (comment)

• lxplus901 for AVX512
• Pick event:

edmCopyPickMerge outputFile=pickevents.root eventsToProcess=367131:196942831 inputFiles=/store/data/Run2023C/DisplacedJet/RAW/v1/000/367/131/00000/9f3f571f-6dc9-4bda-a68b-5d1b9a5fc3ac.root

• Run:

cmsDriver.py step2 --conditions auto:run3_hlt_relval --data --datatier FEVTDEBUGHLT --era Run3_2023 --eventcontent FEVTDEBUGHLT --filein file:pickevents.root --fileout file:step2.root --nStreams 4 --nThreads 8 --number -1 --process reHLT --python_filename step_2_cfg.py --step L1REPACK:Full,HLT:@relval2024 --accelerators cpu

[cmsbuild]

A new Pull Request was created by @valsdav for CMSSW_14_0_X.

It involves the following packages:

• RecoTauTag/HLTProducers (hlt)
• RecoTauTag/RecoTau (reconstruction)

@cmsbuild, @Martin-Grunewald, @mandrenguyen, @mmusich, @jfernan2 can you please review it and eventually sign? Thanks.
@mbluj, @missirol, @azotz, @silviodonato this is something you requested to watch as well.
@sextonkennedy, @antoniovilela, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[valsdav]

assign ml
type tau

[cmsbuild]

New categories assigned: ml

@valsdav,@wpmccormack you have been requested to review this Pull request/Issue and eventually sign? Thanks

[Martin-Grunewald]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dde31f/38246/summary.html
COMMIT: 127750c
CMSSW: CMSSW_14_0_X_2024-03-18-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44456/38246/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 83 lines from the logs
• Reco comparison results: 40 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3346212
• DQMHistoTests: Total failures: 3
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3346187
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 205 log files, 166 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[jfernan2]

+1

[malbouis]

urgent

Since the failure that is fixed here is affecting also the Prompt processing (as mentioned in https://cms-talk.web.cern.ch/t/update-t0-skim-config-for-2024-pp-collision/36794/5 ) we would like to have it in a new 14_0_X release as soon as possible.

[cmsbuild]

Pull request #44456 was updated. @cmsbuild, @jfernan2, @wpmccormack, @mandrenguyen, @Martin-Grunewald, @valsdav, @mmusich can you please check and sign again.

[cmsbuild]

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dde31f/38262/summary.html
COMMIT: 3de18b4
CMSSW: CMSSW_14_0_X_2024-03-19-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44456/38262/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

Comparison Summary

Summary:

• You potentially added 109 lines to the logs
• Reco comparison results: 41 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3346212
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3346190
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 205 log files, 166 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[Martin-Grunewald]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dde31f/38267/summary.html
COMMIT: 3de18b4
CMSSW: CMSSW_14_0_X_2024-03-19-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44456/38267/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 98 lines to the logs
• Reco comparison results: 37 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3346212
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3346190
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 205 log files, 166 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[mmusich]

+hlt

[valsdav]

+ml

[jfernan2]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_14_0_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_14_1_X is complete. This pull request will now be reviewed by the release team before it's merged. @rappoccio, @antoniovilela, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

[antoniovilela]

+1