Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rivet 4.0.0 and YODA 2.0.0 #9111

Merged
merged 5 commits into from
Jul 2, 2024
Merged

Conversation

mseidel42
Copy link
Contributor

@mseidel42 mseidel42 commented Apr 3, 2024

Requires HepMC3, HDF5, and ONNX now.

Announcement at https://rivet.hepforge.org/

CMSSW PR coming -> cms-sw/cmssw#44603

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 3, 2024

A new Pull Request was created by @mseidel42 for branch IB/CMSSW_14_1_X/master.

@smuzaffar, @iarspider, @cmsbuild, @aandvalenzuela can you please review it and eventually sign? Thanks.
@rappoccio, @antoniovilela, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 3, 2024

cms-bot internal usage

@mseidel42
Copy link
Contributor Author

needs cms-sw/cmssw#44603

@smuzaffar
Copy link
Contributor

smuzaffar commented Apr 3, 2024

test parameters:

@smuzaffar
Copy link
Contributor

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 3, 2024

-1

Failed Tests: UnitTests RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c03f46/38582/summary.html
COMMIT: b89d739
CMSSW: CMSSW_14_1_X_2024-04-03-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9111/38582/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c03f46/38582/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c03f46/38582/git-merge-result

Unit Tests

I found 2 errors in the following unit tests:

---> test test-genParticles2HepMC had ERRORS
---> test test-rivet-plot had ERRORS

RelVals

----- Begin Fatal Exception 03-Apr-2024 21:26:15 CEST-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 2 stream: 1
   [1] Running path 'NANOEDMAODSIMoutput_step'
   [2] Prefetching for module PoolOutputModule/'NANOEDMAODSIMoutput'
   [3] Calling method for module SimpleGenEventFlatTableProducer/'genTable'
   Additional Info:
      [a] Fatal Root Error: @SUB=TClass::LoadClassInfo
no interpreter information for class GenEventInfoProduct is available even though it has a TClass initialization routine.

----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 03-Apr-2024 21:38:34 CEST-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 3 stream: 1
   [1] Running path 'NANOEDMAODSIMoutput_step'
   [2] Prefetching for module PoolOutputModule/'NANOEDMAODSIMoutput'
   [3] Calling method for module SimpleGenEventFlatTableProducer/'genTable'
   Additional Info:
      [a] Fatal Root Error: @SUB=TInterpreter::AutoParse
Error parsing payload code for class GenEventInfoProduct with content:

#line 1 "SimDataFormatsGeneratorProducts_xr dictionary payload"

#ifndef CMS_DICT_IMPL
  #define CMS_DICT_IMPL 1
#endif
#ifndef _REENTRANT
  #define _REENTRANT 1
#endif
#ifndef GNUSOURCE
  #define GNUSOURCE 1
#endif
#ifndef __STRICT_ANSI__
  #define __STRICT_ANSI__ 1
#endif
#ifndef GNU_GCC
  #define GNU_GCC 1
#endif
#ifndef _GNU_SOURCE
  #define _GNU_SOURCE 1
#endif
#ifndef TBB_USE_GLIBCXX_VERSION
  #define TBB_USE_GLIBCXX_VERSION 120301
#endif
#ifndef TBB_SUPPRESS_DEPRECATED_MESSAGES
  #define TBB_SUPPRESS_DEPRECATED_MESSAGES 1
#endif
#ifndef TBB_PREVIEW_RESUMABLE_TASKS
  #define TBB_PREVIEW_RESUMABLE_TASKS 1
#endif
#ifndef TBB_PREVIEW_TASK_GROUP_EXTENSIONS
  #define TBB_PREVIEW_TASK_GROUP_EXTENSIONS 1
#endif
#ifndef BOOST_SPIRIT_THREADSAFE
  #define BOOST_SPIRIT_THREADSAFE 1
#endif
#ifndef PHOENIX_THREADSAFE
  #define PHOENIX_THREADSAFE 1
#endif
#ifndef BOOST_MATH_DISABLE_STD_FPCLASSIFY
  #define BOOST_MATH_DISABLE_STD_FPCLASSIFY 1
#endif
#ifndef BOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX
  #define BOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX 1
#endif
#ifndef CMSSW_GIT_HASH
  #define CMSSW_GIT_HASH "CMSSW_14_1_X_2024-04-03-1100"
#endif
#ifndef PROJECT_NAME
  #define PROJECT_NAME "CMSSW"
#endif
#ifndef PROJECT_VERSION
  #define PROJECT_VERSION "CMSSW_14_1_X_2024-04-03-1100"
#endif
#ifndef CMSSW_REFLEX_DICT
  #define CMSSW_REFLEX_DICT 1
#endif

#define _BACKWARD_BACKWARD_WARNING_H
// Inline headers
#include <utility>
#include <vector>
#include <map>
#include <set>

#include "DataFormats/Common/interface/Wrapper.h"
#include "DataFormats/Common/interface/RefVector.h"

#include "SimDataFormats/GeneratorProducts/interface/LHEEventProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/LHERunInfoProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/LHEXMLStringProduct.h"

#include "SimDataFormats/GeneratorProducts/interface/HepMCProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/HepMC3Product.h"

#include "SimDataFormats/GeneratorProducts/interface/GenRunInfoProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/GenFilterInfo.h"
#include "SimDataFormats/GeneratorProducts/interface/GenEventInfoProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/GenEventInfoProduct3.h"
#include "SimDataFormats/GeneratorProducts/interface/GenLumiInfoProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/GenLumiInfoHeader.h"
#include "SimDataFormats/GeneratorProducts/interface/ExternalGeneratorLumiInfo.h"
#include "SimDataFormats/GeneratorProducts/interface/ExternalGeneratorEventInfo.h"

#include <HepMC/GenRanges.h>

//needed for backward compatibility between HepMC 2.06.xx and 2.05.yy
namespace hepmc_rootio {
  void add_to_particles_in(HepMC::GenVertex*, HepMC::GenParticle*);
  void clear_particles_in(HepMC::GenVertex*);

  inline void weightcontainer_set_default_names(unsigned int n,
                                                std::map<std::string, HepMC::WeightContainer::size_type>& names) {
    std::ostringstream name;
    for (HepMC::WeightContainer::size_type count = 0; count < n; ++count) {
      name.str(std::string());
      name << count;
      names[name.str()] = count;
    }
  }
}  // namespace hepmc_rootio

#undef  _BACKWARD_BACKWARD_WARNING_H


----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 03-Apr-2024 21:45:34 CEST-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'dqmoffline_2_step'
   [2] Prefetching for module NanoAODDQM/'nanoDQMMC'
   [3] Calling method for module SimpleGenEventFlatTableProducer/'genTable'
   Additional Info:
      [a] Fatal Root Error: @SUB=TInterpreter::AutoParse
Error parsing payload code for class GenEventInfoProduct with content:

#line 1 "SimDataFormatsGeneratorProducts_xr dictionary payload"

#ifndef CMS_DICT_IMPL
  #define CMS_DICT_IMPL 1
#endif
#ifndef _REENTRANT
  #define _REENTRANT 1
#endif
#ifndef GNUSOURCE
  #define GNUSOURCE 1
#endif
#ifndef __STRICT_ANSI__
  #define __STRICT_ANSI__ 1
#endif
#ifndef GNU_GCC
  #define GNU_GCC 1
#endif
#ifndef _GNU_SOURCE
  #define _GNU_SOURCE 1
#endif
#ifndef TBB_USE_GLIBCXX_VERSION
  #define TBB_USE_GLIBCXX_VERSION 120301
#endif
#ifndef TBB_SUPPRESS_DEPRECATED_MESSAGES
  #define TBB_SUPPRESS_DEPRECATED_MESSAGES 1
#endif
#ifndef TBB_PREVIEW_RESUMABLE_TASKS
  #define TBB_PREVIEW_RESUMABLE_TASKS 1
#endif
#ifndef TBB_PREVIEW_TASK_GROUP_EXTENSIONS
  #define TBB_PREVIEW_TASK_GROUP_EXTENSIONS 1
#endif
#ifndef BOOST_SPIRIT_THREADSAFE
  #define BOOST_SPIRIT_THREADSAFE 1
#endif
#ifndef PHOENIX_THREADSAFE
  #define PHOENIX_THREADSAFE 1
#endif
#ifndef BOOST_MATH_DISABLE_STD_FPCLASSIFY
  #define BOOST_MATH_DISABLE_STD_FPCLASSIFY 1
#endif
#ifndef BOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX
  #define BOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX 1
#endif
#ifndef CMSSW_GIT_HASH
  #define CMSSW_GIT_HASH "CMSSW_14_1_X_2024-04-03-1100"
#endif
#ifndef PROJECT_NAME
  #define PROJECT_NAME "CMSSW"
#endif
#ifndef PROJECT_VERSION
  #define PROJECT_VERSION "CMSSW_14_1_X_2024-04-03-1100"
#endif
#ifndef CMSSW_REFLEX_DICT
  #define CMSSW_REFLEX_DICT 1
#endif

#define _BACKWARD_BACKWARD_WARNING_H
// Inline headers
#include <utility>
#include <vector>
#include <map>
#include <set>

#include "DataFormats/Common/interface/Wrapper.h"
#include "DataFormats/Common/interface/RefVector.h"

#include "SimDataFormats/GeneratorProducts/interface/LHEEventProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/LHERunInfoProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/LHEXMLStringProduct.h"

#include "SimDataFormats/GeneratorProducts/interface/HepMCProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/HepMC3Product.h"

#include "SimDataFormats/GeneratorProducts/interface/GenRunInfoProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/GenFilterInfo.h"
#include "SimDataFormats/GeneratorProducts/interface/GenEventInfoProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/GenEventInfoProduct3.h"
#include "SimDataFormats/GeneratorProducts/interface/GenLumiInfoProduct.h"
#include "SimDataFormats/GeneratorProducts/interface/GenLumiInfoHeader.h"
#include "SimDataFormats/GeneratorProducts/interface/ExternalGeneratorLumiInfo.h"
#include "SimDataFormats/GeneratorProducts/interface/ExternalGeneratorEventInfo.h"

#include <HepMC/GenRanges.h>

//needed for backward compatibility between HepMC 2.06.xx and 2.05.yy
namespace hepmc_rootio {
  void add_to_particles_in(HepMC::GenVertex*, HepMC::GenParticle*);
  void clear_particles_in(HepMC::GenVertex*);

  inline void weightcontainer_set_default_names(unsigned int n,
                                                std::map<std::string, HepMC::WeightContainer::size_type>& names) {
    std::ostringstream name;
    for (HepMC::WeightContainer::size_type count = 0; count < n; ++count) {
      name.str(std::string());
      name << count;
      names[name.str()] = count;
    }
  }
}  // namespace hepmc_rootio

#undef  _BACKWARD_BACKWARD_WARNING_H


----- End Fatal Exception -------------------------------------------------
Expand to see more relval errors ...

RelVals-INPUT

  • 13234.013234.0_TTbar_14TeV+2021FS/step2_TTbar_14TeV+2021FS.log
  • 14034.014034.0_TTbar_14TeV+2023FS/step2_TTbar_14TeV+2023FS.log
  • 13434.013434.0_TTbar_14TeV+2021FSPU/step2_TTbar_14TeV+2021FSPU.log
Expand to see more relval errors ...

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 4, 2024

Pull request #9111 was updated.

@mseidel42
Copy link
Contributor Author

These might be dictionary related errors but they seem transient? See discusison in cms-sw/cmssw#44603 (comment)

Any ideas? :/

@smuzaffar
Copy link
Contributor

smuzaffar commented Apr 4, 2024

@mseidel42 , in the relval logs ( e.g. 135.4 ) I see errors like

In file included from SimDataFormatsGeneratorProducts_xr dictionary payload:72:
/cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9111/38582/CMSSW_14_1_X_2024-04-03-1100/src/SimDataFormats/GeneratorProducts/interface/HepMC3Product.h:11:10: fatal error: 'HepMC3/GenEvent.h' file not found
#include <HepMC3/GenEvent.h>

could it be that we are missing hepmc3 dependency somewhere in cmssw BuildFile

@smuzaffar
Copy link
Contributor

or we now have HepMC3/GenEvent.h part of root dictionaries and we need to update the hepmc3 toolfile to have <runtime name="ROOT_INCLUDE_PATH" value="$INCLUDE" type="path"/> so that rootcling can find hepmc3 headers at rutime

@mseidel42
Copy link
Contributor Author

Ah, just that one line or

  <runtime name="ROOT_INCLUDE_PATH"  value="$INCLUDE" type="path"/>
  <use name="root_cxxdefaults"/>

as in hepmc_headers.xml?

I added hepmc3 to GeneratorInterface/RivetInterface/BuildFile.xml, and Mikhail had it added to SimDataFormats/GeneratorProducts/BuildFile.xml already before :)

@smuzaffar
Copy link
Contributor

just <runtime name="ROOT_INCLUDE_PATH" value="$INCLUDE" type="path"/> should be enough

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 4, 2024

Pull request #9111 was updated.

@smuzaffar
Copy link
Contributor

please test

@mseidel42
Copy link
Contributor Author

Hi Matti, thanks for checking!

I see there are GenEvent::read/write_data() functions, I will try those for passing the event as GenEventData between CMSSW modules.

@mseidel42
Copy link
Contributor Author

Updated cms-sw/cmssw#44603 so that the HepMC3Product now contains GenEventData instead of GenEvent. My local tests look fine...

@smuzaffar
Copy link
Contributor

please test

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c03f46/39583/summary.html
COMMIT: 6863d1f
CMSSW: CMSSW_14_1_X_2024-05-28-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9111/39583/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c03f46/39583/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c03f46/39583/git-merge-result

Comparison Summary

Summary:

@smuzaffar
Copy link
Contributor

please test

@smuzaffar
Copy link
Contributor

please test for el8_aarch64_gcc12

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c03f46/39608/summary.html
COMMIT: 6863d1f
CMSSW: CMSSW_14_1_X_2024-05-28-2300/el8_aarch64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9111/39608/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c03f46/39608/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c03f46/39608/git-merge-result

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c03f46/39607/summary.html
COMMIT: 6863d1f
CMSSW: CMSSW_14_1_X_2024-05-29-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9111/39607/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c03f46/39607/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c03f46/39607/git-merge-result

Comparison Summary

Summary:

  • You potentially added 6 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 14 differences found in the comparisons
  • DQMHistoTests: Total files compared: 48
  • DQMHistoTests: Total histograms compared: 3338862
  • DQMHistoTests: Total failures: 8663
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3330179
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
  • Checked 202 log files, 165 edm output root files, 48 DQM output files
  • TriggerResults: no differences found

@smuzaffar
Copy link
Contributor

please test

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c03f46/39627/summary.html
COMMIT: 6863d1f
CMSSW: CMSSW_14_1_X_2024-05-30-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9111/39627/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 1 lines to the logs
  • Reco comparison results: 15 differences found in the comparisons
  • DQMHistoTests: Total files compared: 48
  • DQMHistoTests: Total histograms compared: 3338862
  • DQMHistoTests: Total failures: 9
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3338833
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
  • Checked 202 log files, 165 edm output root files, 48 DQM output files
  • TriggerResults: no differences found

@mseidel42
Copy link
Contributor Author

Hi, would it be possible to proceed with this PR?

Matti did not have any additional comments in the corresponding cms-sw/cmssw#44603

@smuzaffar
Copy link
Contributor

lets test cms-sw/cmssw#44603 and once that is approved/merged then we merge this PR too

@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 2, 2024

REMINDER @rappoccio, @sextonkennedy, @antoniovilela: This PR was tested with cms-sw/cmssw#44603, please check if they should be merged together

@smuzaffar
Copy link
Contributor

cms-sw/cmssw#44603 has been merged so this should also get in to IB otherwise we will have broken IBs

@smuzaffar smuzaffar merged commit 81b5d6d into cms-sw:IB/CMSSW_14_1_X/master Jul 2, 2024
17 checks passed
@mseidel42
Copy link
Contributor Author

Right, thanks! 🙂

@smuzaffar
Copy link
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants