Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Graph Definition #558

Merged
merged 60 commits into from
Aug 14, 2023
Merged

Graph Definition #558

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
ae4bada
added GraphDefinition, EdgeDefinition
RasmusOrsoe Jun 26, 2023
08cd6b4
refactor of Detector
RasmusOrsoe Jun 26, 2023
238324c
add KNNGraph
RasmusOrsoe Jun 27, 2023
ea52b8d
polish
RasmusOrsoe Jun 28, 2023
f7e93b2
polish
RasmusOrsoe Jun 28, 2023
8a1cace
replace Detector with GraphDefinition for StandardModel
RasmusOrsoe Jun 28, 2023
ccf9d32
Simplify Dataset, polish
RasmusOrsoe Jun 28, 2023
b4b1023
Restructure
RasmusOrsoe Jun 28, 2023
c18e5e1
simplify imports
RasmusOrsoe Jun 28, 2023
99d9b51
simplify imports
RasmusOrsoe Jun 28, 2023
9b9434d
remove redundant import
RasmusOrsoe Jun 28, 2023
8984a01
refactor of promethus detector class
RasmusOrsoe Jun 28, 2023
d5c8ade
refactor of promethus detector class
RasmusOrsoe Jun 28, 2023
6d15019
refactor training example without configs
RasmusOrsoe Jun 28, 2023
fa30515
polish
RasmusOrsoe Jul 18, 2023
4a994bc
polish
RasmusOrsoe Jul 18, 2023
0a21d75
revert
RasmusOrsoe Jul 18, 2023
56474ca
Merge branch 'main' of https://github.com/RasmusOrsoe/graphnet
RasmusOrsoe Jul 18, 2023
50e6afe
revert
RasmusOrsoe Jul 18, 2023
469d396
Merge branch 'main' of https://github.com/RasmusOrsoe/graphnet
RasmusOrsoe Jul 18, 2023
ce248de
merge_conflict_fix
RasmusOrsoe Jul 18, 2023
118b580
Update tito example
RasmusOrsoe Jul 18, 2023
639bf3a
update ensemble dataset example
RasmusOrsoe Jul 18, 2023
a22a794
mypy ignore Detector
RasmusOrsoe Jul 18, 2023
3e01dcd
mypy ignore detector
RasmusOrsoe Jul 18, 2023
fed3900
update dataloader unit tests
RasmusOrsoe Jul 18, 2023
c417ce8
update dataset unit tests, pipeline
RasmusOrsoe Jul 18, 2023
6627d62
update icetray examples
RasmusOrsoe Jul 18, 2023
365c55b
update pipeline example
RasmusOrsoe Jul 18, 2023
1de129a
update dataset config unit test
RasmusOrsoe Jul 18, 2023
53a1861
update dataset config unit test
RasmusOrsoe Jul 18, 2023
378ea8e
update task unit test
RasmusOrsoe Jul 18, 2023
f22c986
update model config unit test
RasmusOrsoe Jul 18, 2023
5110a91
update model config unit test
RasmusOrsoe Jul 18, 2023
e7c9a11
update dataset unit tests
RasmusOrsoe Jul 18, 2023
5dc1298
mypy..
RasmusOrsoe Jul 18, 2023
c510ba0
add relative path for test_data_sqlite
RasmusOrsoe Jul 27, 2023
dc1485d
update path to dataset config
RasmusOrsoe Jul 27, 2023
41da3ab
Remove print statement in 01_train_model.py
RasmusOrsoe Jul 27, 2023
fac6794
remove print statement in graph_definition
RasmusOrsoe Jul 27, 2023
5924d2f
add typo corrections from Andreas
RasmusOrsoe Jul 27, 2023
f7a3aca
delete graph builders
RasmusOrsoe Jul 28, 2023
2d0b5cf
fix paraenthesis
RasmusOrsoe Jul 28, 2023
a2e0508
update IceCubeKaggle detector to new format
RasmusOrsoe Jul 28, 2023
e72678d
Reformat IceCubeDeepCore to new format
RasmusOrsoe Jul 28, 2023
1fb67cc
Update IceCubeUpgrade detector to new format
RasmusOrsoe Jul 28, 2023
0e26587
remove IceCubeUpgrade_V2
RasmusOrsoe Jul 28, 2023
a7216e4
remove cross-inheritance of IceCube Detector classes
RasmusOrsoe Jul 28, 2023
dcd00c5
remove redundant imports
RasmusOrsoe Jul 28, 2023
cec8c48
add GraphDefinition to deployment modules
RasmusOrsoe Jul 28, 2023
cab8c74
add deletion of has_torch_package
RasmusOrsoe Aug 10, 2023
9eb948a
fix comment in model_config.py
RasmusOrsoe Aug 10, 2023
5f1c5fe
test default value of graph_definition dtype
RasmusOrsoe Aug 10, 2023
10e2eda
delete if statement to handle None dtype arg
RasmusOrsoe Aug 10, 2023
af94365
remove property dec. for feature_map in Detector
RasmusOrsoe Aug 10, 2023
df68aa3
update graphnet overview in getting_started.md
RasmusOrsoe Aug 10, 2023
76140f2
Update Dataset and Dataloader section of getting_started.md
RasmusOrsoe Aug 10, 2023
f20f2ed
update example DatasetConfig in getting_started.md
RasmusOrsoe Aug 10, 2023
f174cf3
final update of getting_started.md
RasmusOrsoe Aug 10, 2023
3fb96c8
update
RasmusOrsoe Aug 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
247 changes: 149 additions & 98 deletions GETTING_STARTED.md
View file
Open in desktop

Large diffs are not rendered by default.

13 changes: 13 additions & 0 deletions configs/datasets/dev_lvl7_robustness_muon_neutrino_0000.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,17 @@
path: /groups/icecube/asogaard/data/example/dev_lvl7_robustness_muon_neutrino_0000.db
Copy link
Collaborator

@AMHermansen AMHermansen Jul 24, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that the path is an absolute path instead of a relative path from the graphnet repository. This might be more elegantly dealt with in a separate PR, since it was like this before as well.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I've looked through the code base and I actually cannot find anything that depends on this config file, so I wonder why we still have it. Perhaps it's worth deleting.

graph_definition:
arguments:
columns: [0, 1, 2]
detector:
arguments: {}
class_name: IceCube86
dtype: null
nb_nearest_neighbours: 8
node_definition:
arguments: {}
class_name: NodesAsPulses
node_feature_names: [dom_x, dom_y, dom_z, dom_time, charge, rde, pmt_area]
class_name: KNNGraph
pulsemaps:
- SRTTWOfflinePulsesDC
features:
Expand Down
58 changes: 33 additions & 25 deletions configs/datasets/test_data_sqlite.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,34 @@
path: $GRAPHNET/data/tests/sqlite/oscNext_genie_level7_v02/oscNext_genie_level7_v02_first_5_frames.db
pulsemaps:
- SRTInIcePulses
features:
- dom_x
- dom_y
- dom_z
- dom_time
- charge
- rde
- pmt_area
truth:
- energy
- position_x
- position_y
- position_z
- azimuth
- zenith
- pid
- elasticity
- sim_type
- interaction_type
features: [sensor_pos_x, sensor_pos_y, sensor_pos_z, t]
graph_definition:
arguments:
columns: [0, 1, 2]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the naming of this "columns" arguments (which I understand as the columns included in the "distance" calculation of your edge definition) becomes very vague. At first glance it hard to tell what this feature does. Something like "edge_defining_columns" would be more descriptive I think.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that the name of that argument is not unambiguous, but I also fail to find better alternatives that are not very long. The name of this argument is the same as what we used to call it in the KNNGraphBuilder (see here) and I do think that the doc string for this argument (see here) is pretty clear. So even though it's a bit challenging to understand what that argument does when one reads the config file, I think we should keep it as-is and refer users to the docs instead (which is the intended usage anyway). Is that OK with you? @Aske-Rosted

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Completely fine.

detector:
arguments: {}
class_name: Prometheus
dtype: null
nb_nearest_neighbours: 8
node_definition:
arguments: {}
class_name: NodesAsPulses
node_feature_names: [sensor_pos_x, sensor_pos_y, sensor_pos_z, t]
class_name: KNNGraph
index_column: event_no
truth_table: truth
seed: 21
selection: null
loss_weight_column: null
loss_weight_default_value: null
loss_weight_table: null
node_truth: null
node_truth_table: null
path: $GRAPHNET/data/examples/sqlite/prometheus/prometheus-events.db
pulsemaps: total
seed: null
selection: null
string_selection: null
truth: [injection_energy, injection_type, injection_interaction_type, injection_zenith,
injection_azimuth, injection_bjorkenx, injection_bjorkeny, injection_position_x,
injection_position_y, injection_position_z, injection_column_depth, primary_lepton_1_type,
primary_hadron_1_type, primary_lepton_1_position_x, primary_lepton_1_position_y,
primary_lepton_1_position_z, primary_hadron_1_position_x, primary_hadron_1_position_y,
primary_hadron_1_position_z, primary_lepton_1_direction_theta, primary_lepton_1_direction_phi,
primary_hadron_1_direction_theta, primary_hadron_1_direction_phi, primary_lepton_1_energy,
primary_hadron_1_energy, total_energy]
truth_table: mc_truth
13 changes: 13 additions & 0 deletions configs/datasets/training_classification_example_data_sqlite.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,17 @@
path: $GRAPHNET/data/examples/sqlite/prometheus/prometheus-events.db
graph_definition:
arguments:
columns: [0, 1, 2]
detector:
arguments: {}
class_name: Prometheus
dtype: null
nb_nearest_neighbours: 8
node_definition:
arguments: {}
class_name: NodesAsPulses
node_feature_names: [sensor_pos_x, sensor_pos_y, sensor_pos_z, t]
class_name: KNNGraph
pulsemaps:
- total
features:
Expand Down
13 changes: 13 additions & 0 deletions configs/datasets/training_example_data_parquet.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,17 @@
path: $GRAPHNET/data/examples/parquet/prometheus/prometheus-events.parquet
graph_definition:
arguments:
columns: [0, 1, 2]
detector:
arguments: {}
class_name: Prometheus
dtype: null
nb_nearest_neighbours: 8
node_definition:
arguments: {}
class_name: NodesAsPulses
node_feature_names: [sensor_pos_x, sensor_pos_y, sensor_pos_z, t]
class_name: KNNGraph
pulsemaps:
- total
features:
Expand Down
13 changes: 13 additions & 0 deletions configs/datasets/training_example_data_sqlite.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,17 @@
path: $GRAPHNET/data/examples/sqlite/prometheus/prometheus-events.db
graph_definition:
arguments:
columns: [0, 1, 2]
detector:
arguments: {}
class_name: Prometheus
dtype: null
nb_nearest_neighbours: 8
node_definition:
arguments: {}
class_name: NodesAsPulses
node_feature_names: [sensor_pos_x, sensor_pos_y, sensor_pos_z, t]
class_name: KNNGraph
pulsemaps:
- total
features:
Expand Down
36 changes: 22 additions & 14 deletions configs/models/dynedge_PID_classification_example.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,4 @@
arguments:
coarsening: null
detector:
ModelConfig:
arguments:
graph_builder:
ModelConfig:
arguments: {columns: null, nb_nearest_neighbours: 8}
class_name: KNNGraphBuilder
scalers: null
class_name: Prometheus
gnn:
ModelConfig:
arguments:
Expand All @@ -21,11 +11,29 @@ arguments:
post_processing_layer_sizes: null
readout_layer_sizes: null
class_name: DynEdge
graph_definition:
ModelConfig:
arguments:
columns: [0, 1, 2]
detector:
ModelConfig:
arguments: {}
class_name: Prometheus
dtype: null
nb_nearest_neighbours: 8
node_definition:
ModelConfig:
arguments: {}
class_name: NodesAsPulses
node_feature_names: [sensor_pos_x, sensor_pos_y, sensor_pos_z, t]
class_name: KNNGraph
optimizer_class: '!class torch.optim.adam Adam'
optimizer_kwargs: {eps: 1e-03, lr: 1e-05}
scheduler_class: '!class torch.optim.lr_scheduler ReduceLROnPlateau'
scheduler_config: {frequency: 1, monitor: val_loss}
scheduler_kwargs: {patience: 1}
optimizer_kwargs: {eps: 0.001, lr: 0.001}
scheduler_class: '!class graphnet.training.callbacks PiecewiseLinearLR'
scheduler_config: {interval: step}
scheduler_kwargs:
factors: [0.01, 1, 0.01]
milestones: [0, 20.0, 80]
tasks:
- ModelConfig:
arguments:
Expand Down
44 changes: 0 additions & 44 deletions configs/models/dynedge_energy_example.yml
View file
Open in desktop

This file was deleted.

24 changes: 17 additions & 7 deletions configs/models/dynedge_position_custom_scaling_example.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,24 @@
arguments:
coarsening: null
detector:
graph_definition:
ModelConfig:
arguments:
graph_builder:
detector:
ModelConfig:
arguments: {columns: null, nb_nearest_neighbours: 8}
class_name: KNNGraphBuilder
scalers: null
class_name: IceCubeDeepCore
arguments: {}
class_name: Prometheus
dtype: null
edge_definition:
ModelConfig:
arguments:
columns: [0, 1, 2]
nb_nearest_neighbours: 8
class_name: KNNEdges
node_definition:
ModelConfig:
arguments: {}
class_name: NodesAsPulses
node_feature_names: null
class_name: KNNGraph
gnn:
ModelConfig:
arguments:
Expand Down
20 changes: 13 additions & 7 deletions configs/models/example_direction_reconstruction_model.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,20 @@
arguments:
coarsening: null
detector:
graph_definition:
ModelConfig:
arguments:
graph_builder:
columns: [0, 1, 2]
detector:
ModelConfig:
arguments: {columns: null, nb_nearest_neighbours: 8}
class_name: KNNGraphBuilder
scalers: null
class_name: Prometheus
arguments: {}
class_name: Prometheus
dtype: null
nb_nearest_neighbours: 8
node_definition:
ModelConfig:
arguments: {}
class_name: NodesAsPulses
node_feature_names: [sensor_pos_x, sensor_pos_y, sensor_pos_z, t]
class_name: KNNGraph
gnn:
ModelConfig:
arguments:
Expand Down
39 changes: 24 additions & 15 deletions configs/models/example_energy_reconstruction_model.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,4 @@
arguments:
coarsening: null
detector:
ModelConfig:
arguments:
graph_builder:
ModelConfig:
arguments: {columns: null, nb_nearest_neighbours: 8}
class_name: KNNGraphBuilder
scalers: null
class_name: Prometheus
gnn:
ModelConfig:
arguments:
Expand All @@ -21,11 +11,29 @@ arguments:
post_processing_layer_sizes: null
readout_layer_sizes: null
class_name: DynEdge
graph_definition:
ModelConfig:
arguments:
columns: [0, 1, 2]
detector:
ModelConfig:
arguments: {}
class_name: Prometheus
dtype: null
nb_nearest_neighbours: 8
node_definition:
ModelConfig:
arguments: {}
class_name: NodesAsPulses
node_feature_names: [sensor_pos_x, sensor_pos_y, sensor_pos_z, t]
class_name: KNNGraph
optimizer_class: '!class torch.optim.adam Adam'
optimizer_kwargs: {eps: 0.001, lr: 1e-05}
scheduler_class: '!class torch.optim.lr_scheduler ReduceLROnPlateau'
scheduler_config: {frequency: 1, monitor: val_loss}
scheduler_kwargs: {patience: 5}
optimizer_kwargs: {eps: 0.001, lr: 0.001}
scheduler_class: '!class graphnet.training.callbacks PiecewiseLinearLR'
scheduler_config: {interval: step}
scheduler_kwargs:
factors: [0.01, 1, 0.01]
milestones: [0, 20.0, 80]
tasks:
- ModelConfig:
arguments:
Expand All @@ -35,8 +43,9 @@ arguments:
arguments: {}
class_name: LogCoshLoss
loss_weight: null
prediction_labels: null
target_labels: total_energy
transform_inference: null
transform_inference: '!lambda x: torch.pow(10,x)'
transform_prediction_and_target: '!lambda x: torch.log10(x)'
transform_support: null
transform_target: null
Expand Down
26 changes: 16 additions & 10 deletions configs/models/example_vertex_position_reconstruction_model.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,4 @@
arguments:
coarsening: null
detector:
ModelConfig:
arguments:
graph_builder:
ModelConfig:
arguments: {columns: null, nb_nearest_neighbours: 8}
class_name: KNNGraphBuilder
scalers: null
class_name: Prometheus
gnn:
ModelConfig:
arguments:
Expand All @@ -21,6 +11,22 @@ arguments:
post_processing_layer_sizes: null
readout_layer_sizes: null
class_name: DynEdge
graph_definition:
ModelConfig:
arguments:
columns: [0, 1, 2]
detector:
ModelConfig:
arguments: {}
class_name: Prometheus
dtype: null
nb_nearest_neighbours: 8
node_definition:
ModelConfig:
arguments: {}
class_name: NodesAsPulses
node_feature_names: [sensor_pos_x, sensor_pos_y, sensor_pos_z, t]
class_name: KNNGraph
optimizer_class: '!class torch.optim.adam Adam'
optimizer_kwargs: {eps: 0.001, lr: 0.001}
scheduler_class: '!class graphnet.training.callbacks PiecewiseLinearLR'
Expand Down
8 changes: 3 additions & 5 deletions examples/01_icetray/02_compare_sqlite_and_parquet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,9 @@

from graphnet.constants import EXAMPLE_OUTPUT_DIR, TEST_DATA_DIR
from graphnet.data.constants import FEATURES, TRUTH
from graphnet.data.sqlite import (
SQLiteDataConverter,
SQLiteDataset,
)
from graphnet.data.parquet import ParquetDataConverter, ParquetDataset
from graphnet.data.sqlite import SQLiteDataConverter
from graphnet.data.parquet import ParquetDataConverter
from graphnet.data.dataset import SQLiteDataset, ParquetDataset
from graphnet.data.extractors import (
I3FeatureExtractorIceCube86,
I3TruthExtractor,
Expand Down
Loading