Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

True Positive and Negative rates in multiclosure weighted fits #2258

Open
wants to merge 23 commits into
base: master
Choose a base branch
from
Open

True Positive and Negative rates in multiclosure weighted fits #2258

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
bcbd7c5
added code for computation of TPR and TNR in a multiclosure test
comane Jan 17, 2025
ea56508
added docs and TODO comments
comane Jan 17, 2025
16faf88
changed structure of tpr and tnr dataframe
comane Jan 18, 2025
aa4abcc
added Generator typing and use plotutils wrapper of matplotlib
comane Jan 18, 2025
b969ad0
use api to get dataset label
comane Jan 21, 2025
31973e9
added all mutliclosure nsigma modules
comane Jan 22, 2025
838b511
added multiclosure nsigma modules to __init__ file
comane Jan 22, 2025
c06fa51
added docstrings to functions and moved some functions to helpers
comane Jan 22, 2025
bf0be1c
added plotting function of prob inconsist consist
comane Jan 22, 2025
cc6b78d
added function for computation of probability of consistent/inconsist…
comane Jan 23, 2025
d178508
added docs to module
comane Jan 23, 2025
cac64dc
moved helper functions to helpers module
comane Jan 23, 2025
d0a7109
change plots with z_alpha as x-label
comane Jan 24, 2025
edc4f7f
changed labels as in paper draft
comane Feb 13, 2025
63e5cd5
inverted labels
comane Mar 7, 2025
7e61f62
changed labels and added different linestyle
comane Mar 8, 2025
b0a3ac3
removed unused module
comane Mar 19, 2025
cbdedc5
added runcard to examples
comane Mar 19, 2025
32bfe2a
import from validphys rather than using sys
comane Mar 19, 2025
24891be
use consistent notation with 2503.17447 paper
comane Mar 25, 2025
060907f
use consistent notation with 2503.17447 paper
comane Mar 25, 2025
b46f020
removed hardcoded title
comane Mar 25, 2025
6d31329
import CommonData from nnpdf_data
comane Mar 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
114 changes: 114 additions & 0 deletions validphys2/examples/true_pos_neg_closuretests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
meta:
title: Inconsistency classification probabilities
author: Lazy Person
keywords: [nsigma, chi2, multiclosure test, inconsistent]

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add perhaps a comment saying that the commented dataset are those to reproduce the paper.

weighted_dataset: HERACOMBNCEP575

dataspecs:
- speclabel: "DIS Inconsistent fits, weighted HERA 575"
fits:
- {id: 241227_dis_ict_lam0_weighted_hera575_fs_828872, label: "DIS weighted HERA 575, fs: 1"}
- {id: 241227_dis_ict_lam0_weighted_hera575_fs_239494, label: "DIS weighted HERA 575, fs: 2"}
- {id: 241227_dis_ict_lam0_weighted_hera575_fs_444227, label: "DIS weighted HERA 575, fs: 3"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_623856, label: "DIS weighted HERA 575, fs: 4"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_227492, label: "DIS weighted HERA 575, fs: 5"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_745924, label: "DIS weighted HERA 575, fs: 6"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_724387, label: "DIS weighted HERA 575, fs: 7"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_78994, label: "DIS weighted HERA 575, fs: 8"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_96000, label: "DIS weighted HERA 575, fs: 9"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_997788, label: "DIS weighted HERA 575, fs: 10"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_143862, label: "DIS weighted HERA 575, fs: 11"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_852153, label: "DIS weighted HERA 575, fs: 12"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_219029, label: "DIS weighted HERA 575, fs: 13"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_261213, label: "DIS weighted HERA 575, fs: 14"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_193336, label: "DIS weighted HERA 575, fs: 15"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_352812, label: "DIS weighted HERA 575, fs: 16"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_944815, label: "DIS weighted HERA 575, fs: 17"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_847399, label: "DIS weighted HERA 575, fs: 18"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_562031, label: "DIS weighted HERA 575, fs: 19"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_220528, label: "DIS weighted HERA 575, fs: 20"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_30137, label: "DIS weighted HERA 575, fs: 21"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_680245, label: "DIS weighted HERA 575, fs: 22"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_331227, label: "DIS weighted HERA 575, fs: 23"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_592002, label: "DIS weighted HERA 575, fs: 24"}
# - {id: 241227_dis_ict_lam0_weighted_hera575_fs_955177, label: "DIS weighted HERA 575, fs: 25"}


- speclabel: "DIS Inconsistent fits"
fits:
- {id: 8_6_2023_21_55_4_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_828872, label: "DIS lambda 0, fs: 1"}
- {id: 8_6_2023_21_38_13_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_239494, label: "DIS lambda 0, fs: 2"}
- {id: 8_6_2023_21_41_39_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_444227, label: "DIS lambda 0, fs: 3"}
# - {id: 8_6_2023_21_42_26_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_623856, label: "DIS lambda 0, fs: 4"}
# - {id: 8_6_2023_21_43_13_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_227492, label: "DIS lambda 0, fs: 5"}
# - {id: 8_6_2023_21_44_1_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_745924, label: "DIS lambda 0, fs: 6"}
# - {id: 8_6_2023_21_44_50_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_724387, label: "DIS lambda 0, fs: 7"}
# - {id: 8_6_2023_21_45_39_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_78994, label: "DIS lambda 0, fs: 8"}
# - {id: 8_6_2023_21_46_26_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_96000, label: "DIS lambda 0, fs: 9"}
# - {id: 8_6_2023_21_47_14_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_997788, label: "DIS lambda 0, fs: 10"}
# - {id: 8_6_2023_21_48_55_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_143862, label: "DIS lambda 0, fs: 11"}
# - {id: 8_6_2023_21_48_5_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_852153, label: "DIS lambda 0, fs: 12"}
# - {id: 8_6_2023_21_49_44_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_219029, label: "DIS lambda 0, fs: 13"}
# - {id: 8_6_2023_21_50_32_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_261213, label: "DIS lambda 0, fs: 14"}
# - {id: 8_6_2023_21_51_23_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_193336, label: "DIS lambda 0, fs: 15"}
# - {id: 8_6_2023_21_52_11_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_352812, label: "DIS lambda 0, fs: 16"}
# - {id: 8_6_2023_21_53_47_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_944815, label: "DIS lambda 0, fs: 17"}
# - {id: 8_6_2023_21_56_2_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_847399, label: "DIS lambda 0, fs: 18"}
# - {id: 8_6_2023_21_56_52_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_562031, label: "DIS lambda 0, fs: 19"}
# - {id: 8_6_2023_21_57_55_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_220528, label: "DIS lambda 0, fs: 20"}
# - {id: 8_6_2023_21_59_2_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_30137, label: "DIS lambda 0, fs: 21"}
# - {id: 8_6_2023_22_0_10_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_680245, label: "DIS lambda 0, fs: 22"}
# - {id: 8_6_2023_22_1_12_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_331227, label: "DIS lambda 0, fs: 23"}
# - {id: 8_6_2023_22_2_12_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_592002, label: "DIS lambda 0, fs: 24"}
# - {id: 8_6_2023_22_3_6_mnc_dis_pt1_type2_inconsistent_HERANC_commit_a6f0b1539_filterseed_955177, label: "DIS lambda 0, fs: 25"}

# - speclabel: "DIS Inconsistent fits, weighted NMC"
# fits:
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_828872, label: "DIS weighted NMC, fs 1"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_239494, label: "DIS weighted NMC, fs 2"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_444227, label: "DIS weighted NMC, fs 3"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_623856, label: "DIS weighted NMC, fs 4"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_227492, label: "DIS weighted NMC, fs 5"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_745924, label: "DIS weighted NMC, fs 6"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_724387, label: "DIS weighted NMC, fs 7"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_78994, label: "DIS weighted NMC, fs 8"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_96000, label: "DIS weighted NMC, fs 9"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_997788, label: "DIS weighted NMC, fs 10"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_143862, label: "DIS weighted NMC, fs 11"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_852153, label: "DIS weighted NMC, fs 12"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_219029, label: "DIS weighted NMC, fs 13"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_261213, label: "DIS weighted NMC, fs 14"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_193336, label: "DIS weighted NMC, fs 15"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_352812, label: "DIS weighted NMC, fs 16"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_944815, label: "DIS weighted NMC, fs 17"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_847399, label: "DIS weighted NMC, fs 18"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_562031, label: "DIS weighted NMC, fs 19"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_220528, label: "DIS weighted NMC, fs 20"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_30137, label: "DIS weighted NMC, fs 21"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_680245, label: "DIS weighted NMC, fs 22"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_331227, label: "DIS weighted NMC, fs 23"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_592002, label: "DIS weighted NMC, fs 24"}
# - {id: 241227_dis_ict_lam0_weighted_NMC_fs_955177, label: "DIS weighted NMC, fs 25"}

use_cuts: internal
use_fitcommondata: true
use_weights_in_covmat: false

template_text: |
## Probabilities

#### All sets Probabilities
{@plot_all_sets@}

{@plot_1_minus_all_sets@}

#### Prob of being inconsistent
{@plot_probability_inconsistent@}

#### Prob of being consistent
{@plot_probability_consistent@}


actions_:
- report(main=True)
3 changes: 3 additions & 0 deletions validphys2/src/validphys/closuretest/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,6 @@
from validphys.closuretest.multiclosure_preprocessing import *
from validphys.closuretest.multiclosure_pseudodata import *
from validphys.closuretest.inconsistent_closuretest.multiclosure_inconsistent_output import *
from validphys.closuretest.multiclosure_nsigma_helpers import *
from validphys.closuretest.multiclosure_nsigma import *
from validphys.closuretest.multiclosure_nsigma_output import *
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need this? It would be great to only import the functions that are actually needed. We have to live with those that are already there, but right now if I import closuretest and try to autocomplete I get 301 possibilities, some of them are not really useful (like closuretest.np for numpy...)

Loading
Loading