Implement TH2Poly in DQM Services for HGCal DQM #41932

ywkao · 2023-06-12T15:51:27Z

PR description:

This PR introduces a new type of DQM MonitorElement, TH2Poly, for HGCal DQM in the future. This feature allows a display of polygonal histograms on the CMS DQM GUI. As a demonstration, a wafer map can be displayed like the screenshot here [1].

TH2Poly is a 2D histogram class inherited from TH2. Polygonal bins, defined by TGraph, can be loaded using the AddBin() method. After setting up the polygonal bins, a TH2Poly object can store information through Fill() or SetBinContent().

A workflow for creating polygonal histograms looks like this:
DQM Service -> DQM EDAnalyzer -> CMS DQM GUI

An implementation of TH2Poly in DQM Service and MonitorElement is necessary to display the polygonal histograms. It involves updates on two repositories: cmssw and dqmgui_prod. A pull request is created in the dqmgui_prod repository [2] with a relevant issue reported in this link [3], which is about setting up a CMS DQM GUI with the new feature.

PR validation:

The workflow and the implementation have been tested: (a) From this feature branch, monitor elements of TH2Poly can be stored in a DQM root file [4]. (b) The DQM root file can be uploaded to a CMS DQM GUI, which is built following the steps noted in this issue [3]. Polygonal maps can be displayed on the DQM GUI, as demonstrated in [1].

[1] https://ykao.web.cern.ch/ykao/raw_data_handling/hgcal_dqm_gui/screenshot_demo_th2poly_wafermap.png
[2] cms-DQM/dqmgui_prod#14
[3] cms-DQM/dqmgui_prod#13
[4] A DQM root file containing demo polygonal maps: /afs/cern.ch/work/y/ykao/public/example_HGCAL_DQM/DQM_V0001_HGCAL_R000123469.root

cmsbuild · 2023-06-12T15:59:08Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41932/35885

This PR adds an extra 44KB to repository

cmsbuild · 2023-06-12T15:59:32Z

A new Pull Request was created by @ywkao for master.

It involves the following packages:

DQMServices/Core (dqm)
DataFormats/Histograms (dqm, core)

@smuzaffar, @Dr15Jones, @makortel, @nothingface0, @emanueleusai, @cmsbuild, @pmandrik, @syuvivida, @tjavaid, @micsucmed, @rvenditti can you please review it and eventually sign? Thanks.
@missirol, @barvic, @rovere this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

makortel

I think in addition the following places need to be amended accordingly

cmssw/DQMServices/FwkIO/plugins/DQMRootOutputModule.cc

Lines 250 to 253 in e18c96d

    
           static TreeHelperBase* makeHelper(unsigned int iTypeIndex, TTree* iTree, std::string* iFullNameBufferPtr) { 
        
             switch (iTypeIndex) { 
        
               case kIntIndex: 
        
                 return new IntTreeHelper(iTree, iFullNameBufferPtr);

cmssw/DQMServices/FwkIO/plugins/DQMRootSource.cc

Lines 465 to 468 in e18c96d

    
             m_treeReaders[kTProfileIndex].reset(new TreeObjectReader<TProfile>(MonitorElementData::Kind::TPROFILE, m_rescope)); 
        
             m_treeReaders[kTProfile2DIndex].reset( 
        
                 new TreeObjectReader<TProfile2D>(MonitorElementData::Kind::TPROFILE2D, m_rescope)); 
        
           }

How about adding some tests to make sure all the I/O components work properly? (I don't remember if there are already any such tests that could be easily amended)

makortel · 2023-06-12T19:41:55Z

DQMServices/Core/interface/DQMStore.h

+                                 double lowY,
+                                 double highY,
+                                 FUNC onbooking = NOOP()) {
+        return bookME(name, MonitorElementData::Kind::TH2F, [=]() {


Shouldn't this be

Suggested change

return bookME(name, MonitorElementData::Kind::TH2F, [=]() {

return bookME(name, MonitorElementData::Kind::TH2Poly, [=]() {

?

Indeed, this should be corrected. I will update it in a commit accordingly.

makortel · 2023-06-12T19:43:04Z

DQMServices/Core/src/MonitorElement.cc

+  /// set polygon bin (TH2Poly)
+  void MonitorElement::addBin(TGraph *graph) {
+    auto access = this->accessMut();
+    static_cast<TH2Poly *>(accessRootObject(access, __PRETTY_FUNCTION__, 2))->AddBin(graph);


What if the ROOT object is not TH2Poly? The other similar functions throw an exception via the incompatible() function.

Indeed, incompatible() function is necessary. This block of code will be modified as follows:

- static_cast<TH2Poly *>(accessRootObject(access, __PRETTY_FUNCTION__, 2))->AddBin(graph); + if (kind() == Kind::TH2Poly) { + static_cast<TH2Poly *>(accessRootObject(access, __PRETTY_FUNCTION__, 2))->AddBin(graph); + } else { + incompatible(__PRETTY_FUNCTION__); + }

makortel · 2023-06-12T19:44:31Z

DataFormats/Histograms/interface/MonitorElementCollection.h

@@ -140,7 +140,8 @@ struct MonitorElementData {
    TH2I = 0x23,
    TH3F = 0x30,
    TPROFILE = 0x40,
-    TPROFILE2D = 0x41
+    TPROFILE2D = 0x41,
+    TH2Poly = 0x60


Why such a large gap between the previous value (0x41)?

I guess I kept thinking of hexagons for a wafer map and then put a number starting with six. We can assign 0x24 for TH2Poly if it is feasible.

+ TH2Poly = 0x24, TH3F = 0x30, TPROFILE = 0x40, - TPROFILE2D = 0x41, - TH2Poly = 0x60 + TPROFILE2D = 0x41

emanueleusai · 2023-06-13T03:13:04Z

type hgcal

emanueleusai · 2023-06-13T03:13:13Z

please test

cmsbuild · 2023-06-13T07:05:42Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-03d614/33112/summary.html
COMMIT: 5fa5e4e
CMSSW: CMSSW_13_2_X_2023-06-12-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41932/33112/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 15 lines to the logs
Reco comparison results: 59 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3190910
DQMHistoTests: Total failures: 2477
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3188411
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 207 log files, 159 edm output root files, 48 DQM output files
TriggerResults: no differences found

cmsbuild · 2023-06-13T16:49:38Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41932/35914

This PR adds an extra 40KB to repository

makortel · 2024-08-07T22:01:40Z

From #41932 (comment) at least ywkao@002f7cb looks potentially useful (if not necessary) for general usability of TH2Poly.

cmsbuild · 2024-08-08T01:43:53Z

+1

Size: This PR adds an extra 12KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-03d614/40826/summary.html
COMMIT: 14e856f
CMSSW: CMSSW_14_1_X_2024-08-07-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41932/40826/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 2 lines to the logs
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3423977
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3423957
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
Checked 196 log files, 165 edm output root files, 45 DQM output files
TriggerResults: no differences found

mmusich · 2024-08-09T09:31:53Z

at least ywkao@002f7cb looks potentially useful (if not necessary) for general usability of TH2Poly.

if I am not mistaken ywkao@002f7cb still doesn't compile out-of-the-box in the latest CMSSW IB.

makortel · 2024-08-09T14:27:57Z

at least ywkao@002f7cb looks potentially useful (if not necessary) for general usability of TH2Poly.

if I am not mistaken ywkao@002f7cb still doesn't compile out-of-the-box in the latest CMSSW IB.

Ok. I checked the ROOT version we have in 14_1_X, and the copy operations for TH2Poly (root-project/root#15223).

Could @cms-sw/dqm-l2 comment how useful this PR would be without the developments along ywkao@002f7cb ?

Or should the basis of the update be

we have successfully rebased the branch on CMSSW_14_1_ROOT632_X_2024-04-08-2300 and found no compilation errors. You can inspect the rebased version at this link:

https://github.com/ywkao/cmssw/commits/hgcal-dqm_with_th2poly-14_1_ROOT632_X_2024-04-08-2300/

?

ywkao · 2024-08-20T09:58:34Z

Hi @makortel and @mmusich,

Thank you for bringing this up. Indeed, I missed a few commits. I confirmed that the method mergeProduct() for TH2Poly does not produce compilation error, as checked in this branch:

https://github.com/ywkao/cmssw/commits/dev_th2poly_MEtoEDMFormat
(tested with CMSSW_14_1_ROOT632_X_2024-08-18-2300)

ywkao · 2024-08-20T10:16:59Z

The additional commits in dev_th2poly_MEtoEDMFormat patch the functionality of TH2Poly in DQMServices/Core and DQMServices/Components. If we want to follow the same implementation as TH1I and TH2I in PR #37665, I can work on it this Friday. A list of missing parts is shown below.

DQMServices

Demo/test
FwkIO/plugins
StreamerIO/plugins

DataFormats/Histograms

src/classes_def.xml

ywkao · 2024-08-23T14:06:43Z

Hi @makortel and @mmusich, the missing parts are patched in the same branch, dev_th2poly_MEtoEDMFormat, except for the script DQMServices/Demo/test/runtests.sh. When running the tests, segmentation faults associated with TH2Poly::GetNumberOfBins() and Add() will occur, as shown in [1] produced by the command [2].

I still need to think how to address the issue. I will investigate the issue early next week. If you have any suggestions or comments, I would appreciate your input.

[1] https://ykao.web.cern.ch/ykao/raw_data_handling/pull_request_cmssw/log_20240823.txt
[2] command in DQMServices/Demo/test/runtests.sh

cmsRun ${SCRAM_TEST_PATH}/run_harvesters_cfg.py inputFiles=part1.root inputFiles=part2.root inputFiles=part3.root inputFiles=part4.root outfile=merged.root nomodules=True

makortel · 2024-08-23T14:32:12Z

It seems to me the call to TH2Poly::Add() comes from

cmssw/DQMServices/FwkIO/plugins/DQMRootSource.cc

Line 115 in 87c28ba

original->Add(toAdd);

via

cmssw/DQMServices/FwkIO/plugins/DQMRootSource.cc

Line 204 in 87c28ba

DQMMergeHelper::mergeTogether(existing->getTH1(), m_buffer);

but from just browsing the code I didn't spot anything obviously wrong.

cmsbuild · 2024-08-27T08:08:57Z

Milestone for this pull request has been moved to CMSSW_14_2_X. Please open a backport if it should also go in to CMSSW_14_1_X.

ywkao · 2024-08-29T15:17:21Z

Hi @makortel and @mmusich,

Thank you for your input last Friday. I apologize for the delay in getting back to you.

In fact, TH2Poly::Add() takes two arguments instead of one, as shown below (definition here)

Bool_t TH2Poly::Add(const TH1 * h1, Double_t c1)

So, L115 might cause some problems. However, the following modification does not resolve the segmentation fault after compilation. The error messages are the same.

- 115     original->Add(toAdd);
+ 115     original->Add(toAdd, 1.);

The situation seems similar to the implementation for mergeProduct() in this commit.
ywkao@64a870d#diff-6871d509700385eda0bc39736385861211ce0b8dfa759860e60ee3ad8dc005aeR397-R402

Commands with harvesters turned off also generated the same errors:

cmsRun ${SCRAM_TEST_PATH}/run_harvesters_cfg.py nomodules=1 inputFiles=part1.root inputFiles=part2.root inputFiles=part3.root inputFiles=part4.root outfile=merged.root

I also suspected that TH2Poly objects might need to be set up with AddBins() before adding other histograms, but I do not see a particular error message when testing a simple macro.

Let me know if you have any ideas on this issue.

ywkao · 2024-09-03T08:32:22Z

Hi @makortel and @mmusich,

Thanks to @hqucms, we found that the following commands in root -b in cmssw root will cause a crash. We tested with v6.32.04.

TH2Poly h;
h.GetNumberOfBins();

ywkao · 2024-09-03T15:29:50Z

We would like to propose bypassing the TH2Poly when merging the monitor elements.

+204          if(existing->kind() == MonitorElementData::Kind::TH2Poly) return;
205          DQMMergeHelper::mergeTogether(existing->getTH1(), m_buffer);

makortel · 2024-09-05T19:31:54Z

Thanks to @hqucms, we found that the following commands in root -b in cmssw root will cause a crash. We tested with v6.32.04.
TH2Poly h;
h.GetNumberOfBins();

I'd suggest to report this problem to ROOT team e.g. by opening an issue in https://github.com/root-project/root/. Also FYI @pcanal

hqucms · 2024-09-05T23:02:37Z

Thanks to @hqucms, we found that the following commands in root -b in cmssw root will cause a crash. We tested with v6.32.04.
TH2Poly h;
h.GetNumberOfBins();
I'd suggest to report this problem to ROOT team e.g. by opening an issue in https://github.com/root-project/root/. Also FYI @pcanal

I have made a pull request to ROOT: root-project/root#16370

ywkao · 2024-10-22T14:26:10Z

Thanks to @hqucms, we found that the following commands in root -b in cmssw root will cause a crash. We tested with v6.32.04.
TH2Poly h;
h.GetNumberOfBins();
I'd suggest to report this problem to ROOT team e.g. by opening an issue in https://github.com/root-project/root/. Also FYI @pcanal
I have made a pull request to ROOT: root-project/root#16370

Hi @makortel, the PR root-project/root#16370 has been merged. We need core software people to help update the ROOT version in CMSSW to incorporate this feature.

makortel · 2024-10-22T14:50:53Z

Hi @makortel, the PR root-project/root#16370 has been merged. We need core software people to help update the ROOT version in CMSSW to incorporate this feature.

That PR was already included in the latest ROOT master IB update (cms-sw/cmsdist#9476), and will be part of the next CMSSW_14_2_ROOT6_X IB.

Could you (or someone else) ask the ROOT PR to be backported to 6.32 and 6.30 branches?

hqucms · 2024-10-23T15:52:07Z

Hi @makortel, the PR root-project/root#16370 has been merged. We need core software people to help update the ROOT version in CMSSW to incorporate this feature.

That PR was already included in the latest ROOT master IB update (cms-sw/cmsdist#9476), and will be part of the next CMSSW_14_2_ROOT6_X IB.

Could you (or someone else) ask the ROOT PR to be backported to 6.32 and 6.30 branches?

The backports to 6.32 and 6.30 have been merged:

Fix segfault when calling GetNumberOfBins() on an empty TH2Poly root-project/root#16734
Fix segfault when calling GetNumberOfBins() on an empty TH2Poly root-project/root#16735

makortel · 2024-10-23T19:01:33Z

Hi @makortel, the PR root-project/root#16370 has been merged. We need core software people to help update the ROOT version in CMSSW to incorporate this feature.

That PR was already included in the latest ROOT master IB update (cms-sw/cmsdist#9476), and will be part of the next CMSSW_14_2_ROOT6_X IB.
Could you (or someone else) ask the ROOT PR to be backported to 6.32 and 6.30 branches?

The backports to 6.32 and 6.30 have been merged:

Fix segfault when calling GetNumberOfBins() on an empty TH2Poly root-project/root#16734

Fix segfault when calling GetNumberOfBins() on an empty TH2Poly root-project/root#16735

@smuzaffar Could we get these PRs included in our default and 6.32 ROOT builds?

smuzaffar · 2024-10-24T09:29:38Z

@smuzaffar Could we get these PRs included in our default and 6.32 ROOT builds?

ROOT6.30: cms-sw/cmsdist#9479
ROOT632: cms-sw/cmsdist#9480

ywkao added 2 commits June 11, 2023 23:31

Implement TH2Poly in DQM Services for HGCal DQM

a5ffeba

Apply changes from code quality checks

5fa5e4e

cmsbuild added this to the CMSSW_13_2_X milestone Jun 12, 2023

cmsbuild added dqm-pending core-pending pending-signatures tests-pending orp-pending code-checks-pending labels Jun 12, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 12, 2023

makortel reviewed Jun 12, 2023

View reviewed changes

cmsbuild added tests-started hgcal and removed tests-pending labels Jun 13, 2023

cmsbuild added tests-approved and removed tests-started labels Jun 13, 2023

Add a patch based on makortel's comments

14e856f

cmsbuild added tests-pending code-checks-pending and removed tests-approved code-checks-approved labels Jun 13, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 13, 2023

cmsbuild added the tests-started label Aug 7, 2024

cmsbuild added tests-approved and removed tests-started labels Aug 8, 2024

cmsbuild modified the milestones: CMSSW_14_1_X, CMSSW_14_2_X Aug 27, 2024

cmsbuild modified the milestones: CMSSW_14_1_X, CMSSW_14_2_X Aug 29, 2024

This was referenced Oct 24, 2024

Updated root to tip of branch v6-30-00-patches cms-sw/cmsdist#9479

Merged

[ROOT632] Updated root to tip of branch v6-32-00-patches cms-sw/cmsdist#9480

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement TH2Poly in DQM Services for HGCal DQM #41932

Implement TH2Poly in DQM Services for HGCal DQM #41932

ywkao commented Jun 12, 2023

cmsbuild commented Jun 12, 2023

cmsbuild commented Jun 12, 2023 •

edited

Loading

makortel left a comment

makortel Jun 12, 2023

ywkao Jun 13, 2023

makortel Jun 12, 2023

ywkao Jun 13, 2023 •

edited

Loading

makortel Jun 12, 2023

ywkao Jun 13, 2023

emanueleusai commented Jun 13, 2023

emanueleusai commented Jun 13, 2023

cmsbuild commented Jun 13, 2023

cmsbuild commented Jun 13, 2023

makortel commented Aug 7, 2024

cmsbuild commented Aug 8, 2024

mmusich commented Aug 9, 2024

makortel commented Aug 9, 2024

ywkao commented Aug 20, 2024

ywkao commented Aug 20, 2024

ywkao commented Aug 23, 2024

makortel commented Aug 23, 2024

cmsbuild commented Aug 27, 2024

ywkao commented Aug 29, 2024

ywkao commented Sep 3, 2024

ywkao commented Sep 3, 2024

makortel commented Sep 5, 2024

hqucms commented Sep 5, 2024 •

edited

Loading

ywkao commented Oct 22, 2024

makortel commented Oct 22, 2024

hqucms commented Oct 23, 2024

makortel commented Oct 23, 2024

smuzaffar commented Oct 24, 2024

	static TreeHelperBase* makeHelper(unsigned int iTypeIndex, TTree* iTree, std::string* iFullNameBufferPtr) {
	switch (iTypeIndex) {
	case kIntIndex:
	return new IntTreeHelper(iTree, iFullNameBufferPtr);

	m_treeReaders[kTProfileIndex].reset(new TreeObjectReader<TProfile>(MonitorElementData::Kind::TPROFILE, m_rescope));
	m_treeReaders[kTProfile2DIndex].reset(
	new TreeObjectReader<TProfile2D>(MonitorElementData::Kind::TPROFILE2D, m_rescope));
	}

	return bookME(name, MonitorElementData::Kind::TH2F, [=]() {
	return bookME(name, MonitorElementData::Kind::TH2Poly, [=]() {

Implement TH2Poly in DQM Services for HGCal DQM #41932

Are you sure you want to change the base?

Implement TH2Poly in DQM Services for HGCal DQM #41932

Conversation

ywkao commented Jun 12, 2023

PR description:

PR validation:

cmsbuild commented Jun 12, 2023

cmsbuild commented Jun 12, 2023 • edited Loading

makortel left a comment

Choose a reason for hiding this comment

makortel Jun 12, 2023

Choose a reason for hiding this comment

ywkao Jun 13, 2023

Choose a reason for hiding this comment

makortel Jun 12, 2023

Choose a reason for hiding this comment

ywkao Jun 13, 2023 • edited Loading

Choose a reason for hiding this comment

makortel Jun 12, 2023

Choose a reason for hiding this comment

ywkao Jun 13, 2023

Choose a reason for hiding this comment

emanueleusai commented Jun 13, 2023

emanueleusai commented Jun 13, 2023

cmsbuild commented Jun 13, 2023

Comparison Summary

cmsbuild commented Jun 13, 2023

makortel commented Aug 7, 2024

cmsbuild commented Aug 8, 2024

Comparison Summary

mmusich commented Aug 9, 2024

makortel commented Aug 9, 2024

ywkao commented Aug 20, 2024

ywkao commented Aug 20, 2024

ywkao commented Aug 23, 2024

makortel commented Aug 23, 2024

cmsbuild commented Aug 27, 2024

ywkao commented Aug 29, 2024

ywkao commented Sep 3, 2024

ywkao commented Sep 3, 2024

makortel commented Sep 5, 2024

hqucms commented Sep 5, 2024 • edited Loading

ywkao commented Oct 22, 2024

makortel commented Oct 22, 2024

hqucms commented Oct 23, 2024

makortel commented Oct 23, 2024

smuzaffar commented Oct 24, 2024

cmsbuild commented Jun 12, 2023 •

edited

Loading

ywkao Jun 13, 2023 •

edited

Loading

hqucms commented Sep 5, 2024 •

edited

Loading