treeDepth-08-08-distributedMEGATraining-32-grayscale.log

Starting the distributed training at Thu  8 Aug 22:57:43 BST 2024
Command running in screen session distributed_MEGAtrain_0_20240808-225743 on gpu03
Command running in screen session distributed_MEGAtrain_1_20240808-225743 on gpu16
Command running in screen session distributed_MEGAtrain_2_20240808-225743 on gpu17
Command running in screen session distributed_MEGAtrain_3_20240808-225743 on gpu11
Command running in screen session distributed_MEGAtrain_4_20240808-225743 on gpu13
Command running in screen session distributed_MEGAtrain_5_20240808-225743 on gpu14
Command running in screen session distributed_MEGAtrain_6_20240808-225743 on gpu23
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
Command running in screen session distributed_MEGAtrain_7_20240808-225743 on gpu12
Log file path is /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/treeDepth-08-08-distributedMEGATraining-32-grayscale.log
Learning rate is now 1e-5 nto 1e-4
Distributed training completed at Thu  8 Aug 22:57:59 BST 2024
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/08/2024 22:58:07 gluefactory INFO] Starting experiment sp+lg_08-08-distributedMEGA-grayscale-32-pretrain
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/08/2024 22:58:14 gluefactory INFO] Starting experiment sp+lg_08-08-distributedMEGA-grayscale-32-pretrain
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/08/20/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
outputs saved to /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_08-08-distributedMEGA-grayscale-32-pretrain/log.txt
[08/08/2024 22:58:28 gluefactory INFO] Will fine-tune from weights of pretrain_lightglue
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:1724: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setti/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
outputs saved to /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_08-08-distributedMEGA-grayscale-32-pretrain/log.txt
[08/08/2024 22:58:32 gluefactory INFO] Will fine-tune from weights of pretrain_lightglue
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:1724: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  init_cp = torch.load(str(init_cp), map_location="cpu")
[08/08/2024 22:58:32 gluefactory INFO] Training in distributed mode with 8 GPUs
[08/08/2024 22:58:34 gluefactory INFO] Using device cuda:0
[08/08/2024 22:58:34 gluefactory.datasets.base_dataset INFO] Creating dataset TreeDepth
[08/08/2024 22:58:34 gluefactory.datasets.treedepth INFO] Initialized TreeDepth dataset with configuration: {'name': 'treedepth', 'num_workers': 1, 'train_batch_size': '???', 'val_batch_size': '???', 'test_batch_size': '???', 'shuffle_training': False, 'batch_size': 4, 'num_threads': 1, 'seed': 0, 'prefetch_factor': 2, 'data_dir': 'syntheticForestData/', 'depth_subpath': 'depthData/', 'image_subpath': 'imageData/', 'info_dir': 'fileLists', 'train_split': 'train_scenes_clean.txt', 'train_num_per_scene': 300, 'val_split': 'valid_scenes_clean.txt', 'val_num_per_scene': None, 'val_pairs': 'valid_pairs.txt', 'test_split': 'test_scenes_clean.txt', 'test_num_per_scene': None, 'test_pairs': None, 'views': 2, 'min_overlap': 0.1, 'max_overlap': 0.7, 'num_overlap_bins': 3, 'sort_by_overlap': False, 'triplet_enforce_overlap': False, 'read_depth': True, 'read_image': True, 'grayscale': False, 'preprocessing': {'resize': None, 'edge_divisible_by': None, 'side': 'long', 'interpolation': 'bilinear', 'align_corners': None, 'antialias': True, 'square_pad': False, 'add_padding_mask': False}, 'p_rotate': 0.0, 'reseed': False, 'load_features': {'do': False, 'path': 'exports/closed-treedepth-rgb-undist-depthr_SP-k2048-nms3/{scene}.h5', 'data_keys': None, 'device': None, 'trainable': False, 'add_data_path': True, 'collate': False, 'scale': ['keypoints', 'lines', 'orig_lines'], 'padding_fn': 'pad_local_features', 'padding_length': 2048, 'numeric_type': 'float32'}}
[08/08/2024 22:58:36 gluefactory.datasets.treedepth INFO] Sampling new items for train with seed 0.
[08/08/2024 22:58:38 gluefactory.datasets.treedepth INFO] Sampling new items for val with seed 0.
[08/08/2024 22:58:38 gluefactory INFO] Training loader has 1159 batches
[08/08/2024 22:58:38 gluefactory INFO] Validation loader has 1435 batches
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/lightglue.py:24: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/depth_matcher.py:40: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/08/2024 22:58:46 gluefactory INFO] Parameters with scaled learning rate:
{}
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:1889: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
  scaler = GradScaler(enabled=args.mixed_precision is not None)
[08/08/2024 22:58:46 gluefactory INFO] Training with mixed_precision=None
[08/08/2024 22:58:46 gluefactory INFO] Starting training with configuration:
data:
  name: treedepth
  train_split: train_scenes_clean.txt
  train_num_per_scene: 300
  val_split: valid_scenes_clean.txt
  val_pairs: valid_pairs.txt
  min_overlap: 0.1
  max_overlap: 0.7
  num_overlap_bins: 3
  read_depth: true
  read_image: true
  batch_size: 32
  num_workers: 4
  load_features:
    do: false
    path: exports/closed-treedepth-rgb-undist-depthr_SP-k2048-nms3/{scene}.h5
    padding_length: 2048
    padding_fn: pad_local_features
model:
  name: two_view_pipeline
  extractor:
    name: gluefactory_nonfree.superpoint
    max_num_keypoints: 512
    force_num_keypoints: true
    detection_threshold: 0.0
    nms_radius: 3
    trainable: false
  ground_truth:
    name: matchers.depth_matcher
    th_positive: 3
    th_negative: 5
    th_epi: 3
  matcher:
    name: matchers.lightglue
    filter_threshold: 0.1
    flash: true
    checkpointed: true
  allow_no_extract: true
train:
  seed: 42
  epochs: 50
  optimizer: adam
  opt_regexp: null
  optimizer_options: {}
  lr: 1.0e-05
  lr_schedule:
    type: exp
    start: 10
    exp_div_10: 45
    on_epoch: true
    factor: 1.0
    options: {}
  lr_scaling:
  - - 100
    - - dampingnet.const
  eval_every_iter: 500
  save_every_iter: 5000
  log_every_iter: 250
  log_grad_every_iter: null
  test_every_epoch: 1
  keep_last_checkpoints: 50
  load_experiment: pretrain_lightglue
  median_metrics: []
  recall_metrics: {}
  pr_metrics: {}
  best_key: loss/total
  dataset_callback_fn: sample_new_items
  dataset_callback_on_val: false
  clip_grad: null
  pr_curves: {}
  plot:
  - 5
  - gluefactory.visualization.visualize_batch.make_match_figures
  submodules: []
benchmarks:
  treeEval1:
    data:
      preprocessing:
        side: long
        resize: 1600
    eval:
      estimator: poselib
      ransac_th: 0.5
  finnEval:
    data:
      preprocessing:
        side: long
        resize: 1600
    eval:
      estimator: poselib
      ransac_th: 0.5
image_mode: null

[08/08/2024 22:58:46 gluefactory INFO] Starting epoch 0
getting last checkpoint pretrain_lightglue
Checkpoint path used is /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/pretrain_lightglue/checkpoint_best_355_backup.tar 
model conf after setting, loaded from checkpoint!!! {'name': 'two_view_pipeline', 'extractor': {'name': 'gluefactory_nonfree.superpoint', 'max_num_keypoints': 512, 'force_num_keypoints': True, 'detection_threshold': 0.0, 'nms_radius': 3, 'trainable': False}, 'ground_truth': {'name': 'matchers.depth_matcher', 'th_positive': 3, 'th_negative': 5, 'th_epi': 3}, 'matcher': {'name': 'matchers.lightglue', 'filter_threshold': 0.1, 'flash': True, 'checkpointed': True}, 'allow_no_extract': True}
the rank is 5 and world size is 8
cuda:0
23 initpy data sets path gluefactory.datasets.treedepth
205 tools.py utils the classes are  [('TreeDepth', <class 'gluefactory.datasets.treedepth.TreeDepth'>)]
136 in treedepth
conf keys: dict_keys(['name', 'num_workers', 'train_batch_size', 'val_batch_size', 'test_batch_size', 'shuffle_training', 'batch_size', 'num_threads', 'seed', 'prefetch_factor', 'data_dir', 'depth_subpath', 'image_subpath', 'info_dir', 'train_split', 'train_num_per_scene', 'val_split', 'val_num_per_scene', 'val_pairs', 'test_split', 'test_num_per_scene', 'test_pairs', 'views', 'min_overlap', 'max_overlap', 'num_overlap_bins', 'sort_by_overlap', 'triplet_enforce_overlap', 'read_depth', 'read_image', 'grayscale', 'preprocessing', 'p_rotate', 'reseed', 'load_features'])
Loaded training dataset: treedepth
calling get_data_loader on dataset object - dataset.get_data_loader('train', distributed=args.distributed, shuffle=False)


scene_lists_path: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/gluefactory/datasets/tartanSceneLists(Full)


root and info /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/data/syntheticForestData fileLists
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!


scene_lists_path: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/gluefactory/datasets/tartanSceneLists(Full)


root and info /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/data/syntheticForestData fileLists
!!!!!sample_new_items: num_per_scene: None
did a random shuffle!!!
205 tools.py utils the classes are  [('TwoViewPipeline', <class 'gluefactory.models.two_view_pipeline.TwoViewPipeline'>)]
205 tools.py utils the classes are  [('SuperPoint', <class 'gluefactory_nonfree.superpoint.SuperPoint'>)]
ModuleNotFoundError: matchers.lightglue
205 tools.py utils the classes are  []
ModuleNotFoundError: matchers.depth_matcher
205 tools.py utils the classes are  [('DepthMatcher', <class 'gluefactory.models.matchers.depth_matcher.DepthMatcher'>)]
93 base_model.py DepthMatcher is trainable
93 base_model.py TwoViewPipeline is trainable




conf.model.name two_view_pipeline 




replaced!! the weights using two_view_pipeline



args distibuted
device ids are [device(type='cuda', index=0)]
after mpodel weight sub in
591 train.py, conf.train.lr_schedule.on_epoch is: True
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:2098: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with autocast(enabled=args.mixed_precision is not None, dtype=mp_dtype):
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py:600: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  return fn(*args, **kwargs)
  epochs: 50
  optimizer: adam
  opt_regexp: null
  optimizer_options: {}
  lr: 1.0e-05
  lr_schedule:
    type: exp
    start: 10
    exp_div_10: 45
    on_epoch: true
    factor: 1.0
    options: {}
  lr_scaling:
  - - 100
    - - dampingnet.const
  eval_every_iter: 500
  save_every_iter: 5000
  log_every_iter: 250
  log_grad_every_iter: null
  test_every_epoch: 1
  keep_last_checkpoints: 50
  load_experiment: pretrain_lightglue
  median_metrics: []
  recall_metrics: {}
  pr_metrics: {}
  best_key: loss/total
  dataset_callback_fn: sample_new_items
  dataset_callback_on_val: false
  clip_grad: null
  pr_curves: {}
  plot:
  - 5
  - gluefactory.visualization.visualize_batch.make_match_figures
  submodules: []
benchmarks:
  treeEval1:
    data:
      preprocessing:
        side: long
        resize: 1600
    eval:
      estimator: poselib
      ransac_th: 0.5
  finnEval:
    data:
      preprocessing:
        side: long
        resize: 1600
    eval:
      estimator: poselib
      ransac_th: 0.5
image_mode: null

[08/08/2024 22:58:46 gluefactory INFO] Starting epoch 0
getting last checkpoint pretrain_lightglue
Checkpoint path used is /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/outputs/training/pretrain_lightglue/checkpoint_best_355_backup.tar 
model conf after setting, loaded from checkpoint!!! {'name': 'two_view_pipeline', 'extractor': {'name': 'gluefactory_nonfree.superpoint', 'max_num_keypoints': 512, 'force_num_keypoints': True, 'detection_threshold': 0.0, 'nms_radius': 3, 'trainable': False}, 'ground_truth': {'name': 'matchers.depth_matcher', 'th_positive': 3, 'th_negative': 5, 'th_epi': 3}, 'matcher': {'name': 'matchers.lightglue', 'filter_threshold': 0.1, 'flash': True, 'checkpointed': True}, 'allow_no_extract': True}
the rank is 6 and world size is 8
cuda:0
23 initpy data sets path gluefactory.datasets.treedepth
205 tools.py utils the classes are  [('TreeDepth', <class 'gluefactory.datasets.treedepth.TreeDepth'>)]
136 in treedepth
conf keys: dict_keys(['name', 'num_workers', 'train_batch_size', 'val_batch_size', 'test_batch_size', 'shuffle_training', 'batch_size', 'num_threads', 'seed', 'prefetch_factor', 'data_dir', 'depth_subpath', 'image_subpath', 'info_dir', 'train_split', 'train_num_per_scene', 'val_split', 'val_num_per_scene', 'val_pairs', 'test_split', 'test_num_per_scene', 'test_pairs', 'views', 'min_overlap', 'max_overlap', 'num_overlap_bins', 'sort_by_overlap', 'triplet_enforce_overlap', 'read_depth', 'read_image', 'grayscale', 'preprocessing', 'p_rotate', 'reseed', 'load_features'])
Loaded training dataset: treedepth
calling get_data_loader on dataset object - dataset.get_data_loader('train', distributed=args.distributed, shuffle=False)


scene_lists_path: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/gluefactory/datasets/tartanSceneLists(Full)


root and info /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/data/syntheticForestData fileLists
!!!!!sample_new_items: num_per_scene: 300
did a random shuffle!!!


scene_lists_path: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/gluefactory/datasets/tartanSceneLists(Full)


root and info /vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/data/syntheticForestData fileLists
!!!!!sample_new_items: num_per_scene: None
did a random shuffle!!!
205 tools.py utils the classes are  [('TwoViewPipeline', <class 'gluefactory.models.two_view_pipeline.TwoViewPipeline'>)]
205 tools.py utils the classes are  [('SuperPoint', <class 'gluefactory_nonfree.superpoint.SuperPoint'>)]
ModuleNotFoundError: matchers.lightglue
205 tools.py utils the classes are  []
ModuleNotFoundError: matchers.depth_matcher
205 tools.py utils the classes are  [('DepthMatcher', <class 'gluefactory.models.matchers.depth_matcher.DepthMatcher'>)]
93 base_model.py DepthMatcher is trainable
93 base_model.py TwoViewPipeline is trainable




conf.model.name two_view_pipeline 




replaced!! the weights using two_view_pipeline



args distibuted
device ids are [device(type='cuda', index=0)]
after mpodel weight sub in
591 train.py, conf.train.lr_schedule.on_epoch is: True
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:2098: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with autocast(enabled=args.mixed_precision is not None, dtype=mp_dtype):
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py:600: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  return fn(*args, **kwargs)
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
[08/08/2024 22:59:17 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_08-08-distributedMEGA-grayscale-32-pretrain/train_sp+lg_08-08-distributedMEGA-grayscale-32-pretrain_intermediate_0_0_20240808_225917.pkl

[08/08/2024 22:59:17 gluefactory INFO] [E 0 | it 0] loss {total 1.110E-01, last 5.126E-02, assignment_nll 5.126E-02, nll_pos 4.874E-02, nll_neg 5.378E-02, num_matchable 5.297E+01, num_unmatchable 6.520E+01, confidence 3.539E-02, row_norm 2.442E-01}
Current iteraiton: 0
Data logged to training DataFrame: {'epoch': 0, 'iteration': 0, 'total': 0.11101330071687698, 'last': 0.05125986784696579, 'assignment_nll': 0.05125986784696579, 'nll_pos': 0.048736780881881714, 'nll_neg': 0.053782954812049866, 'num_matchable': 52.96875, 'num_unmatchable': 65.203125, 'confidence': 0.0353919118642807, 'row_norm': 0.24415363371372223}
line 813in train.py doing validation
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torchmp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:2295: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  results_df = pd.concat([results_df, new_entry], ignore_index=True)
[08/08/2024 23:08:27 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_08-08-distributedMEGA-grayscale-32-pretrain/validation_sp+lg_08-08-distributedMEGA-grayscale-32-pretrain_20240808_230827.pkl

[08/08/2024 23:08:27 gluefactory INFO] [Validation] {match_recall 9.599E-01, match_precision 8.454E-01, accuracy 9.272E-01, average_precision 8.124E-01, loss/total 3.739E-01, loss/last 3.739E-01, loss/assignment_nll 3.739E-01, loss/nll_pos 4.576E-01, loss/nll_neg 2.902E-01, loss/num_matchable 2.006E+02, loss/num_unmatchable 2.620E+02, loss/row_norm 9.657E-01}
[08/08/2024 23:08:27 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[08/08/2024 23:08:28 gluefactory INFO] New best val: loss/total=0.373885192579301
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:2098: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with autocast(enabled=args.mixed_precision is not None, dtype=mp_dtype):
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:2295: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  results_df = pd.concat([results_df, new_entry], ignore_index=True)
[08/08/2024 23:09:18 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_08-08-distributedMEGA-grayscale-32-pretrain/validation_sp+lg_08-08-distributedMEGA-grayscale-32-pretrain_20240808_230918.pkl

[08/08/2024 23:09:18 gluefactory INFO] [Validation] {match_recall 9.599E-01, match_precision 8.454E-01, accuracy 9.272E-01, average_precision 8.124E-01, loss/total 3.739E-01, loss/last 3.739E-01, loss/assignment_nll 3.739E-01, loss/nll_pos 4.576E-01, loss/nll_neg 2.902E-01, loss/num_matchable 2.006E+02, loss/num_unmatchable 2.620E+02, loss/row_norm 9.657E-01}
[08/08/2024 23:09:18 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[08/08/2024 23:09:20 gluefactory INFO] New best val: loss/total=0.37388519170184586
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:2098: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with autocast(enabled=args.mixed_precision is not None, dtype=mp_dtype):
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:2295: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  results_df = pd.concat([results_df, new_entry], ignore_index=True)
[08/08/2024 23:11:41 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_08-08-distributedMEGA-grayscale-32-pretrain/validation_sp+lg_08-08-distributedMEGA-grayscale-32-pretrain_20240808_231141.pkl

[08/08/2024 23:11:41 gluefactory INFO] [Validation] {match_recall 9.599E-01, match_precision 8.454E-01, accuracy 9.272E-01, average_precision 8.124E-01, loss/total 3.739E-01, loss/last 3.739E-01, loss/assignment_nll 3.739E-01, loss/nll_pos 4.576E-01, loss/nll_neg 2.902E-01, loss/num_matchable 2.006E+02, loss/num_unmatchable 2.620E+02, loss/row_norm 9.657E-01}
[08/08/2024 23:11:41 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[08/08/2024 23:11:42 gluefactory INFO] New best val: loss/total=0.3739014555300985
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:2098: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with autocast(enabled=args.mixed_precision is not None, dtype=mp_dtype):
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py:600: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  return fn(*args, **kwargs)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:2295: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  results_df = pd.concat([results_df, new_entry], ignore_index=True)
[08/08/2024 23:12:38 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_08-08-distributedMEGA-grayscale-32-pretrain/validation_sp+lg_08-08-distributedMEGA-grayscale-32-pretrain_20240808_231238.pkl

[08/08/2024 23:12:38 gluefactory INFO] [Validation] {match_recall 9.599E-01, match_precision 8.454E-01, accuracy 9.272E-01, average_precision 8.124E-01, loss/total 3.739E-01, loss/last 3.739E-01, loss/assignment_nll 3.739E-01, loss/nll_pos 4.576E-01, loss/nll_neg 2.902E-01, loss/num_matchable 2.006E+02, loss/num_unmatchable 2.620E+02, loss/row_norm 9.657E-01}
[08/08/2024 23:12:38 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[08/08/2024 23:12:40 gluefactory INFO] New best val: loss/total=0.3738879432246245
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:2098: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with autocast(enabled=args.mixed_precision is not None, dtype=mp_dtype):
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:2295: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  results_df = pd.concat([results_df, new_entry], ignore_index=True)
[08/08/2024 23:27:48 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_08-08-distributedMEGA-grayscale-32-pretrain/validation_sp+lg_08-08-distributedMEGA-grayscale-32-pretrain_20240808_232748.pkl

[08/08/2024 23:27:48 gluefactory INFO] [Validation] {match_recall 9.599E-01, match_precision 8.454E-01, accuracy 9.272E-01, average_precision 8.124E-01, loss/total 3.739E-01, loss/last 3.739E-01, loss/assignment_nll 3.739E-01, loss/nll_pos 4.576E-01, loss/nll_neg 2.902E-01, loss/num_matc/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py:2295: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  results_df = pd.concat([results_df, new_entry], ignore_index=True)
[08/08/2024 23:28:03 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/traipackages/torch/_dynamo/eval_frame.py:600: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  return fn(*args, **kwargs)
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.
  with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
[08/08/2024 23:52:46 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_08-08-distributedMEGA-grayscale-32-pretrain/train_sp+lg_08-08-distributedMEGA-grayscale-32-pretrain_intermediate_0_250_20240808_235246.pkl

[08/08/2024 23:52:46 gluefactory INFO] [E 0 | it 250] loss {total 2.033E-01, last 1.088E-01, assignment_nll 1.088E-01, nll_pos 1.250E-01, nll_neg 9.253E-02, num_matchable 1.479E+02, num_unmatchable 1.504E+02, confidence 5.810E-02, row_norm 5.981E-01}
[08/09/2024 00:17:06 gluefactory INFO] \Training intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_08-08-distributedMEGA-grayscale-32-pretrain/train_sp+lg_08-08-distributedMEGA-grayscale-32-pretrain_intermediate_0_500_20240809_001706.pkl

[08/09/2024 00:17:06 gluefactory INFO] [E 0 | it 500] loss {total 8.422E-02, last 5.142E-02, assignment_nll 5.142E-02, nll_pos 6.917E-02, nll_neg 3.367E-02, num_matchable 6.228E+01, num_unmatchable 5.591E+01, confidence 2.359E-02, row_norm 2.364E-01}
Data logged to validation: {'epoch': 0, 'iteration': 0, 'match_recall': 0.9599463042481851, 'match_precision': 0.8454384348949073, 'accuracy': 0.9272375224774723, 'average_precision': 0.8124219649552468, 'loss/total': 0.3739014555300985, 'loss/last': 0.3739014555300985, 'loss/assignment_nll': 0.3739014555300985, 'loss/nll_pos': 0.4575837137106196, 'loss/nll_neg': 0.2902191973982528, 'loss/num_matchable': 200.5684668989547, 'loss/num_unmatchable': 261.95644599303137, 'loss/row_norm': 0.9657286119377987}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 0.3739014555300985
 best_eval inf
Current iteraiton: 250
Data logged to training DataFrame: {'epoch': 0, 'iteration': 250, 'total': 0.05791632831096649, 'last': 0.030127961188554764, 'assignment_nll': 0.030127961188554764, 'nll_pos': 0.0343434102833271, 'nll_neg': 0.025912512093782425, 'num_matchable': 63.0625, 'num_unmatchable': 58.53125, 'confidence': 0.017821982502937317, 'row_norm': 0.24194461107254028}
Current iteraiton: 500
Data logged to training DataFrame: {'epoch': 0, ': 0, 'iteration': 500, 'total': 0.22896626591682434, 'last': 0.1555422842502594, 'assignment_nll': 0.1555422842502594, 'nll_pos': 0.21333900094032288, 'nll_neg': 0.09774558991193771, 'num_matchable': 102.53125, 'num_unmatchable': 130.953125, 'confidence': 0.05143791064620018, 'row_norm': 0.46255695819854736}
line 813in train.py doing validation
/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/models/matchers/loadedLightGlue.py:25: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  @torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
[08/09/2024 00:28:34 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_08-08-distributedMEGA-grayscale-32-pretrain/validation_sp+lg_08-08-distributedMEGA-grayscale-32-pretrain_20240809_002834.pkl

[08/09/2024 00:28:34 gluefactory INFO] [Validation] {match_recall 9.535E-01, match_precision 8.448E-01, accuracy 9.256E-01, average_precision 8.067E-01, loss/total 3.320E-01, loss/last 3.320E-01, loss/assignment_nll 3.320E-01, loss/nll_pos 4.281E-01, loss/nll_neg 2.360E-01, loss/num_matchable 2.006E+02, loss/num_unmatchable 2.620E+02, loss/row_norm 9.186E-01}
[08/09/2024 00:28:34 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[08/09/2024 00:28:36 gluefactory INFO] New best val: loss/total=0.33202751624926874
[08/09/2024 00:30:40 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_08-08-distributedMEGA-grayscale-32-pretrain/validation_sp+lg_08-08-distributedMEGA-grayscale-32-pretrain_20240809_003040.pkl

[08/09/2024 00:30:40 gluefactory INFO] [Validation] {match_recall 9.535E-01, match_precision 8.448E-01, accuracy 9.256E-01, average_precision 8.068E-01, loss/total 3.320E-01, loss/last 3.320E-01, loss/assignment_nll 3.320E-01, loss/nll_pos 4.281E-01, loss/nll_neg 2.360E-01, loss/num_matchable 2.006E+02, loss/num_unmatchable 2.620E+02, loss/row_norm 9.186E-01}
[08/09/2024 00:30:40 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[08/09/2024 00:30:42 gluefactory INFO] New best val: loss/total=0.33203839344274294
[08/09/2024 00:48:28 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_08-08-distributedMEGA-grayscale-32-pretrain/validation_sp+lg_08-08-distributedMEGA-grayscale-32-pretrain_20240809_004828.pkl

[08/09/2024 00:48:28 gluefactory INFO] [Validation] {match_recall 9.535E-01, match_precision 8.448E-01, accuracy 9.256E-01, average_precision 8.067E-01, loss/total 3.320E-01, loss/last 3.320E-01, loss/assignment_nll 3.320E-01, loss/nll_pos 4.281E-01, loss/nll_neg 2.360E-01, loss/num_matchable 2.006E+02, loss/num_unmatchable 2.620E+02, loss/row_norm 9.186E-01}
[08/09/2024 00:48:28 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[08/09/2024 00:48:30 gluefactory INFO] New best val: loss/total=0.3320306187982343
Data logged to validation: {'epoch': 0, 'iteration': 500, 'match_recall': 0.9535066804819406, 'match_precision': 0.8447882757369649, 'accuracy': 0.9256137585806099, 'average_precision': 0.806733153609864, 'loss/total': 0.33203061920840565, 'loss/last': 0.33203061920840565, 'loss/assignment_nll': 0.33203061920840565, 'loss/nll_pos': 0.42808790531411817, 'loss/nll_neg': 0.23597333210062899, 'loss/num_matchable': 200.56829268292682, 'loss/num_unmatchable': 261.95670731707315, 'loss/row_norm': 0.918632994130098}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 0.33203061920840565
 best_eval 0.373885192579301
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2577, in <module>
    torch.multiprocessing.spawn(
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 282, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 238, in start_processes
    while not context.join():
              ^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 189, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 76, in _wrap
    fn(i, *args)
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2486, in main_worker
    training(rank, conf, output_dir, args, results_df, train_df)
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2132, in training
    torch.distributed.all_reduce(
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 79, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 2293, in all_reduce
    work.wait()
RuntimeError: [/opt/conda/conda-bld/pytorch_1720538439675/work/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 1800000ms for recv operation to complete

Data logged to validation: {'epoch': 0, 'iteration': 500, 'match_recall': 0.9535066804819406, 'match_precision': 0.8447882757369649, 'accuracy': 0.9256137585806099, 'average_precision': 0.806733153609864, 'loss/total': 0.33203061899553193, 'loss/last': 0.33203061899553193, 'loss/assignment_nll': 0.33203061899553193, 'loss/nll_pos': 0.4280879064979038, 'loss/nll_neg': 0.23597333210062899, 'loss/num_matchable': 200.56829268292682, 'loss/num_unmatchable': 261.95670731707315, 'loss/row_norm': 0.918632994130098}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 0.33203061899553193
 best_eval 0.37388519170184586
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2577, in <module>
    torch.multiprocessing.spawn(
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 282, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 238, in start_processes
    while not context.join():
              ^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 189, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 76, in _wrap
    fn(i, *args)
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2486, in main_worker
    training(rank, conf, output_dir, args, results_df, train_df)
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2132, in training
    torch.distributed.all_reduce(
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 79, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 2293, in all_reduce
    work.wait()
RuntimeError: [/opt/conda/conda-bld/pytorch_1720538439675/work/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 1800000ms for recv operation to complete

Data logged to validation: {'epoch': 0, 'iteration': 500, 'match_recall': 0.9535066804819406, 'match_precision': 0.8447882757369649, 'accuracy': 0.9256137585806099, 'average_precision': 0.806733153609864, 'loss/total': 0.33203061899553193, 'loss/last': 0.33203061899553193, 'loss/assignment_nll': 0.33203061899553193, 'loss/nll_pos': 0.4280879064979038, 'loss/nll_neg': 0.23597333210062899, 'loss/num_matchable': 200.56829268292682, 'loss/num_unmatchable': 261.95670731707315, 'loss/row_norm': 0.918632994130098}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 0.33203061899553193
 best_eval 0.37388519170184586
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2577, in <module>
    torch.multiprocessing.spawn(
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 282, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 238, in start_processes
    while not context.join():
              ^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 189, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 76, in _wrap
    fn(i, *args)
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2486, in main_worker
    training(rank, conf, output_dir, args, results_df, train_df)
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2132, in training
    torch.distributed.all_reduce(
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 79, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 2293, in all_reduce
    work.wait()
RuntimeError: [/opt/conda/conda-bld/pytorch_1720538439675/work/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 1800000ms for recv operation to complete

Data logged to validation: {'epoch': 0, 'iteration': 500, 'match_recall': 0.953492382802199, 'match_precision': 0.8447879019929972, 'accuracy': 0.9256190567481809, 'average_precision': 0.8067266925702111, 'loss/total': 0.33202751624926874, 'loss/last': 0.33202751624926874, 'loss/assignment_nll': 0.33202751624926874, 'loss/nll_pos': 0.42807926590901635, 'loss/nll_neg': 0.23597576833794134, 'loss/num_matchable': 200.56829268292682, 'loss/num_unmatchable': 261.95670731707315, 'loss/row_norm': 0.9186302119015817}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 0.33202751624926874
 best_eval 0.37389697020529455
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2577, in <module>
    torch.multiprocessing.spawn(
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 282, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 238, in start_processes
    while not context.join():
              ^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 189, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 76, in _wrap
    fn(i, *args)
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2486, in main_worker
    training(rank, conf, output_dir, args, results_df, train_df)
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2132, in training
    torch.distributed.all_reduce(
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 79, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 2293, in all_reduce
    work.wait()
RuntimeError: [/opt/conda/conda-bld/pytorch_1720538439675/work/third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [146.169.53.218]:47287

Data logged to validation: {'epoch': 0, 'iteration': 500, 'match_recall': 0.9535066804819406, 'match_precision': 0.8447882757369649, 'accuracy': 0.9256137585806099, 'average_precision': 0.806733153609864, 'loss/total': 0.3320306187982343, 'loss/last': 0.3320306187982343, 'loss/assignment_nll': 0.3320306187982343, 'loss/nll_pos': 0.42808790568794525, 'loss/nll_neg': 0.23597333210062899, 'loss/num_matchable': 200.56829268292682, 'loss/num_unmatchable': 261.95670731707315, 'loss/row_norm': 0.918632994130098}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 0.3320306187982343
 best_eval 0.37388519351386856
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2577, in <module>
    torch.multiprocessing.spawn(
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 282, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 238, in start_processes
    while not context.join():
              ^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 189, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 76, in _wrap
    fn(i, *args)
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2486, in main_worker
    training(rank, conf, output_dir, args, results_df, train_df)
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2132, in training
    torch.distributed.all_reduce(
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 79, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 2293, in all_reduce
    work.wait()
RuntimeError: [/opt/conda/conda-bld/pytorch_1720538439675/work/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 1800000ms for recv operation to complete

[08/09/2024 01:14:16 gluefactory INFO] VALIDATION Intermediate DataFrame saved to: /homes/tp4618/Documents/bitbucket/SuperGlueThesis/external/glue-factory/outputs/training/sp+lg_08-08-distributedMEGA-grayscale-32-pretrain/validation_sp+lg_08-08-distributedMEGA-grayscale-32-pretrain_20240809_011416.pkl

[08/09/2024 01:14:16 gluefactory INFO] [Validation] {match_recall 9.535E-01, match_precision 8.448E-01, accuracy 9.256E-01, average_precision 8.067E-01, loss/total 3.320E-01, loss/last 3.320E-01, loss/assignment_nll 3.320E-01, loss/nll_pos 4.281E-01, loss/nll_neg 2.360E-01, loss/num_matchable 2.006E+02, loss/num_unmatchable 2.620E+02, loss/row_norm 9.186E-01}
[08/09/2024 01:14:17 gluefactory.utils.experiments INFO] Saving checkpoint checkpoint_best.tar
[08/09/2024 01:14:19 gluefactory INFO] New best val: loss/total=0.3320306187982343
Data logged to validation: {'epoch': 0, 'iteration': 500, 'match_recall': 0.9535066804819406, 'match_precision': 0.8447882757369649, 'accuracy': 0.9256137585806099, 'average_precision': 0.806733153609864, 'loss/total': 0.3320306187982343, 'loss/last': 0.3320306187982343, 'loss/assignment_nll': 0.3320306187982343, 'loss/nll_pos': 0.42808790568794525, 'loss/nll_neg': 0.23597333210062899, 'loss/num_matchable': 200.56829268292682, 'loss/num_unmatchable': 261.95670731707315, 'loss/row_norm': 0.918632994130098}
comparing to see if it hte best experiment 875 train.py
key used was loss/total
 FROM RESULTS [conf.train.best_key] 0.3320306187982343
 best_eval 0.37388519351386856
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2577, in <module>
    torch.multiprocessing.spawn(
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 282, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 238, in start_processes
    while not context.join():
              ^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 189, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 76, in _wrap
    fn(i, *args)
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2486, in main_worker
    training(rank, conf, output_dir, args, results_df, train_df)
  File "/vol/bitbucket/tp4618/SuperGlueThesis/external/glue-factory/gluefactory/train.py", line 2132, in training
    torch.distributed.all_reduce(
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 79, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/homes/tp4618/Documents/bitbucket/miniconda3/envs/redoGlue/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 2293, in all_reduce
    work.wait()
RuntimeError: [/opt/conda/conda-bld/pytorch_1720538439675/work/third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [146.169.53.226]:33996