Added advanced spatial augmentations and other fixes (#252)

* fix: save best final model * fix date time format * refactor common code * refactor common code * refactor * fix: dataloader size for multi gpu * added BaseModule for freeze batch norm * added BaseModule * updated models with BaseModule * removed avg training time * removed avg training time * fix unit test * added batch norm config to trainer * fix unit test engine * update step trainer * empty cuda cache * fix: ddp training * added mock unit tests (#225) * added mock unit tests (#233) * merge ddp fix to development (#235) * Bump version for DDP fix release * Development (#234) * Add tagged commit trigger for deployment workflow (#214) * Update training config file to comply with the changes in the trainer (#215) * Updated training config file to comply with changes in the trainer * Made changes to default base trainer config as well * Revert "Add tagged commit trigger for deployment workflow (#214)" (#217) This reverts commit 6386fcb. * fix training logs (#219) * fix training logs * setup local rank in ddp * fix step trainer logs * fix: save best final model * fix date time format * refactor common code * refactor common code * refactor * fix: dataloader size for multi gpu * added BaseModule for freeze batch norm * added BaseModule * updated models with BaseModule * removed avg training time * removed avg training time * fix unit test * added batch norm config to trainer * fix unit test engine * update step trainer * empty cuda cache * fix: ddp training * added mock unit tests (#225) * Bump ujson from 1.35 to 5.2.0 in /docs (#226) Bumps [ujson](https://github.com/ultrajson/ultrajson) from 1.35 to 5.2.0. - [Release notes](https://github.com/ultrajson/ultrajson/releases) - [Commits](ultrajson/ultrajson@v1.35...5.2.0) --- updated-dependencies: - dependency-name: ujson dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Loosened requirements (#228) * Loosened requirements (#229) * Bump cookiecutter from 1.7.3 to 2.1.1 in /docs (#230) Bumps [cookiecutter](https://github.com/cookiecutter/cookiecutter) from 1.7.3 to 2.1.1. - [Release notes](https://github.com/cookiecutter/cookiecutter/releases) - [Changelog](https://github.com/cookiecutter/cookiecutter/blob/master/HISTORY.md) - [Commits](cookiecutter/cookiecutter@1.7.3...2.1.1) --- updated-dependencies: - dependency-name: cookiecutter dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump arrow from 0.13.1 to 0.15.1 in /docs (#231) Bumps [arrow](https://github.com/arrow-py/arrow) from 0.13.1 to 0.15.1. - [Release notes](https://github.com/arrow-py/arrow/releases) - [Changelog](https://github.com/arrow-py/arrow/blob/master/CHANGELOG.rst) - [Commits](arrow-py/arrow@0.13.1...0.15.1) --- updated-dependencies: - dependency-name: arrow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump arrow from 0.13.1 to 0.15.1 (#232) Bumps [arrow](https://github.com/arrow-py/arrow) from 0.13.1 to 0.15.1. - [Release notes](https://github.com/arrow-py/arrow/releases) - [Changelog](https://github.com/arrow-py/arrow/blob/master/CHANGELOG.rst) - [Commits](arrow-py/arrow@0.13.1...0.15.1) --- updated-dependencies: - dependency-name: arrow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Version bump * added mock unit tests (#233) Co-authored-by: Neelay Shah <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: NeelayS <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix: unbounded cropsize * fix ddp tensorboard logging * Added Augmentation Operations and unit tests (#237) * Added new augment ops and normalization * Added uni test for augmentation * Changed uni test for augmentation * Augmentation changes * Fixed issue * Added translate func fix and documentation * Made suggested changes and unit tested * Normalize param issue fix * Duplicate removal and norm_param style change * Fix docstrings Co-authored-by: Prajnan Goswami <[email protected]> * fix tensorboard logging multi-gpu * fix start step * update step trainer * :fix unit test * Pwcnet refactor (#242) * PWCNet refactor * update pwcnet config * fix model outputs * fix flow output in evaluation * fix engine unit tests * update multiscale loss * remove total loss logging for step trainer * console output formatting * fix dependency * fix dependency * remove normalization operation from the model * update docstring * update trainer * update flownet C model * update flownet c config * bugfix FlowNetC * fix flownet_c * rearrage transformation sequence * added chromatic augmentations * added chromatic augmentations * added chromatic augmentations * added new spatial transforms to augmentor * bugfix cropsize * toggle transformations * bug fix augmentation * Augmentations (#243) * added default behavior * bugfix: default augmentation behavior * bugfix: chromatic aug * fix noise transform * update trainer * update flownet_c upsampling * update logging in trainer * update ddp training * bugfix distributed training * fix min scale * fix: flow not available in dataset for prediction * sync main (#251) * Bump oauthlib from 3.1.1 to 3.2.1 in /docs (#245) Bumps [oauthlib](https://github.com/oauthlib/oauthlib) from 3.1.1 to 3.2.1. - [Release notes](https://github.com/oauthlib/oauthlib/releases) - [Changelog](https://github.com/oauthlib/oauthlib/blob/master/CHANGELOG.rst) - [Commits](oauthlib/oauthlib@v3.1.1...v3.2.1) --- updated-dependencies: - dependency-name: oauthlib dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump oauthlib from 3.1.1 to 3.2.1 (#244) Bumps [oauthlib](https://github.com/oauthlib/oauthlib) from 3.1.1 to 3.2.1. - [Release notes](https://github.com/oauthlib/oauthlib/releases) - [Changelog](https://github.com/oauthlib/oauthlib/blob/master/CHANGELOG.rst) - [Commits](oauthlib/oauthlib@v3.1.1...v3.2.1) --- updated-dependencies: - dependency-name: oauthlib dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump protobuf from 3.17.3 to 3.18.3 in /docs (#247) Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 3.17.3 to 3.18.3. - [Release notes](https://github.com/protocolbuffers/protobuf/releases) - [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/generate_changelog.py) - [Commits](protocolbuffers/protobuf@v3.17.3...v3.18.3) --- updated-dependencies: - dependency-name: protobuf dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump protobuf from 3.17.3 to 3.18.3 (#246) Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 3.17.3 to 3.18.3. - [Release notes](https://github.com/protocolbuffers/protobuf/releases) - [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/generate_changelog.py) - [Commits](protocolbuffers/protobuf@v3.17.3...v3.18.3) --- updated-dependencies: - dependency-name: protobuf dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix docs errors (#248) * Removed version info from docs config * Fixed autodoc errors * Bump certifi from 2021.5.30 to 2022.12.7 in /docs (#250) Bumps [certifi](https://github.com/certifi/python-certifi) from 2021.5.30 to 2022.12.7. - [Release notes](https://github.com/certifi/python-certifi/releases) - [Commits](certifi/python-certifi@2021.05.30...2022.12.07) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump certifi from 2021.5.30 to 2022.12.7 (#249) Bumps [certifi](https://github.com/certifi/python-certifi) from 2021.5.30 to 2022.12.7. - [Release notes](https://github.com/certifi/python-certifi/releases) - [Commits](certifi/python-certifi@2021.05.30...2022.12.07) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Neelay Shah <[email protected]> * removed chromatic aug * set default augmentation params * decouple flip transform * update aug_params * bugfix: random flip augmentation * bugfix: color transformation * fix engine unit test * fix functional unit tests * update Multiscale loss unit test * fix pre-commit code-style * fix augmentator color params * remove duplicate transformations * update docstring * update docstrings * update docstrings * add Kubric dataset * update READMe: * add link to kubric .tfrecord convert script * update model configs * updated training configs * updated training configs Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: NeelayS <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Devroop Kar <[email protected]> Co-authored-by: Neelay Shah <[email protected]>
neu-vi · Dec 19, 2022 · ad41465 · ad41465
1 parent 1b50c87
commit ad41465
Show file tree

Hide file tree

Showing 56 changed files with 2,267 additions and 729 deletions.
diff --git a/README.md b/README.md
@@ -56,6 +56,7 @@ pip install ezflow
 - [x] [FlyingChairs](https://lmb.informatik.uni-freiburg.de/resources/datasets/FlyingChairs.en.html#flyingchairs)
 - [x] [HD1K](http://hci-benchmark.iwr.uni-heidelberg.de/)
 - [x] [KITTI](http://www.cvlibs.net/datasets/kitti/eval_scene_flow.php?benchmark=flow)
+- [x] [Kubric](https://github.com/google-research/kubric)
 - [x] [MPI Sintel](http://sintel.is.tue.mpg.de/)
 - [x] [SceneFlow Monkaa](https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html)
 - [x] [SceneFlow Driving](https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html)

diff --git a/configs/models/flownet_c.yaml b/configs/models/flownet_c.yaml
@@ -5,11 +5,10 @@ ENCODER:
   CONFIG: [64, 128, 256, 256, 512, 512, 512, 512, 1024, 1024]
   NORM: batch
 SIMILARITY:
-  NAME: CorrelationLayer
-  PAD_SIZE: 10
+  NAME: IterSpatialCorrelationSampler
+  PAD_SIZE: 0
   MAX_DISPLACEMENT: 10
 DECODER:
   NAME: FlowNetConvDecoder
   IN_CHANNELS: 1024
-  CONFIG: [512, 256, 128, 64]
-INTERPOLATE_FLOW: True
+  CONFIG: [512, 256, 128, 64]
diff --git a/configs/models/pwcnet.yaml b/configs/models/pwcnet.yaml
@@ -3,13 +3,11 @@ ENCODER:
   NAME: PyramidEncoder
   IN_CHANNELS: 3
   CONFIG: [16, 32, 64, 96, 128, 196]
-SIMILARITY:
-  NAME: CorrelationLayer
-  PAD_SIZE: 4
-  MAX_DISPLACEMENT: 4
 DECODER:
-  NAME: ConvDecoder
+  NAME: PyramidDecoder
   CONFIG: [128, 128, 96, 64, 32]
   TO_FLOW: True
-INTERPOLATE_FLOW: True
-FLOW_SCALE_FACTOR: 20.0
+  SIMILARITY:
+    PAD_SIZE: 0
+    MAX_DISPLACEMENT: 4
+  FLOW_SCALE_FACTOR: 20.0
diff --git a/configs/models/raft.yaml b/configs/models/raft.yaml
@@ -6,15 +6,15 @@ ENCODER:
     OUT_CHANNELS: 256
     NORM: instance
     P_DROPOUT: 0.0
-    LAYER_CONFIG: [32, 64, 96]
+    LAYER_CONFIG: [64, 96, 128]
     INTERMEDIATE_FEATURES: False
   CONTEXT:
     NAME: BasicEncoder
     IN_CHANNELS: 3
     OUT_CHANNELS: 256
     NORM: batch
     P_DROPOUT: 0.0
-    LAYER_CONFIG: [32, 64, 96]
+    LAYER_CONFIG: [64, 96, 128]
     INTERMEDIATE_FEATURES: False
 HIDDEN_DIM: 128
 CONTEXT_DIM: 128

diff --git a/configs/models/raft_small.yaml b/configs/models/raft_small.yaml
@@ -1,22 +1,30 @@
-_BASE_: "../base.yaml"
-NAME: RAFT_SMALL
+NAME: RAFT
 ENCODER:
   FEATURE: 
-    NAME: BottleneckEncoder
+    NAME: BasicEncoder
     IN_CHANNELS: 3
-    OUT_CHANNELS: 256
+    OUT_CHANNELS: 128
     NORM: instance
     P_DROPOUT: 0.0
     LAYER_CONFIG: [32, 64, 96]
     INTERMEDIATE_FEATURES: False
   CONTEXT:
     NAME: BasicEncoder
     IN_CHANNELS: 3
-    OUT_CHANNELS: 256
+    OUT_CHANNELS: 160
     NORM: batch
     P_DROPOUT: 0.0
     LAYER_CONFIG: [32, 64, 96]
     INTERMEDIATE_FEATURES: False
 HIDDEN_DIM: 96
 CONTEXT_DIM: 64
-CORR_RADIUS: 3
+SIMILARITY:
+  NAME: MutliScalePairwise4DCorr
+  NUM_LEVELS: 4
+DECODER:
+  NAME: RecurrentLookupUpdateBlock
+  INPUT_DIM: 96
+CORR_RADIUS: 4
+CORR_LEVELS: 3
+MIXED_PRECISION: False
+UPDATE_ITERS: 12
diff --git a/configs/trainers/_base_/chairs_baseline.yaml b/configs/trainers/_base_/chairs_baseline.yaml
@@ -0,0 +1,89 @@
+DATA:
+  TRAIN_DATASET:
+    NAME: "flyingchairs"
+    ROOT_DIR: "./Datasets/FlyingChairs_release/data"
+  VAL_DATASET:
+    NAME: "flyingchairs"
+    ROOT_DIR: "./Datasets/FlyingChairs_release/data" 
+  NUM_WORKERS: 4
+  PIN_MEMORY: True
+  APPEND_VALID_MASK: False
+  SHUFFLE: True
+  AUGMENTATION:
+    # Augmentation Settings borrowed from RAFT
+    USE: True
+    PARAMS:
+      TRAINING:
+        COLOR_AUG_PARAMS: {
+          "enabled": True,
+          "asymmetric_color_aug_prob": 0.2, 
+          "brightness": 0.4, 
+          "contrast": 0.4, 
+          "saturation": 0.4, 
+          "hue": 0.15915494309189535
+        }
+        ERASER_AUG_PARAMS: {
+          "enabled": True,
+          "aug_prob": 0.5,
+          "bounds": [50, 100]
+        }
+        NOISE_AUG_PARAMS: {
+          "enabled": False,
+          "aug_prob": 0.5,
+          "noise_std_range": 0.06 
+        }
+        FLIP_AUG_PARAMS: {
+          "enabled": True, 
+          "h_flip_prob": 0.5, 
+          "v_flip_prob": 0.1
+        }
+        SPATIAL_AUG_PARAMS: {
+          "enabled": True,
+          "aug_prob": 0.8, 
+          "stretch_prob": 0.8, 
+          "min_scale": -0.1, 
+          "max_scale": 1.0, 
+          "max_stretch": 0.2, 
+        }
+        ADVANCED_SPATIAL_AUG_PARAMS: {
+          "enabled": False,
+          "scale1": 0.0,
+          "scale2": 0.0,
+          "stretch": 0.0,
+          "rotate": 0.0,
+          "translate": 0.0,
+          "enable_out_of_boundary_crop": False
+        }
+      VALIDATION:
+        SPATIAL_AUG_PARAMS: {"enabled": False}
+        COLOR_AUG_PARAMS: {"enabled": False}
+        ERASER_AUG_PARAMS: {"enabled": False}
+        FLIP_AUG_PARAMS: {"enabled": False}
+        ADVANCED_SPATIAL_AUG_PARAMS : {"enabled": False}
+OPTIMIZER:
+  NAME: AdamW
+  LR: 0.0004
+  PARAMS:
+    weight_decay: 0.0001
+    betas: [0.9, 0.999]
+    eps: 1.e-08
+    amsgrad: False
+GRAD_CLIP: 
+  USE: True
+  VALUE: 1.0
+FREEZE_BATCH_NORM: False
+TARGET_SCALE_FACTOR: 1.0
+MIXED_PRECISION: False
+DEVICE: "0"
+DISTRIBUTED:
+  USE: False
+  WORLD_SIZE: 2
+  BACKEND: nccl
+  MASTER_ADDR: localhost
+  MASTER_PORT: "12355"
+EPOCHS: null
+NUM_STEPS: null
+RESUME_TRAINING:
+  CONSOLIDATED_CKPT: null
+  EPOCHS: null
+  START_EPOCH: null
diff --git a/configs/trainers/_base_/kubric_baseline.yaml b/configs/trainers/_base_/kubric_baseline.yaml
@@ -0,0 +1,87 @@
+DATA:
+  TRAIN_DATASET:
+    NAME: "kubric"
+    ROOT_DIR: "./Datasets/KubricFlow"
+  VAL_DATASET:
+    NAME: "kubric"
+    ROOT_DIR: "./Datasets/KubricFlow" 
+  NUM_WORKERS: 4
+  PIN_MEMORY: True
+  APPEND_VALID_MASK: False
+  SHUFFLE: True
+  AUGMENTATION:
+    USE: True
+    PARAMS:
+      TRAINING:
+        COLOR_AUG_PARAMS: {
+          "enabled": True,
+          "asymmetric_color_aug_prob": 0.2, 
+          "brightness": 0.4, 
+          "contrast": 0.4, 
+          "saturation": 0.4, 
+          "hue": 0.15915494309189535
+        }
+        ERASER_AUG_PARAMS: {
+          "enabled": True,
+          "aug_prob": 0.5,
+          "bounds": [50, 100]
+        }
+        NOISE_AUG_PARAMS: {
+          "enabled": False,
+          "aug_prob": 0.5,
+          "noise_std_range": 0.06 
+        }
+        FLIP_AUG_PARAMS: {
+          "enabled": True, 
+          "h_flip_prob": 0.5, 
+          "v_flip_prob": 0.1
+        }
+        SPATIAL_AUG_PARAMS: {
+          "enabled": True,
+          "aug_prob": 0.8, 
+          "stretch_prob": 0.8, 
+          "min_scale": -0.1, 
+          "max_scale": 1.0, 
+          "max_stretch": 0.2, 
+        }
+        ADVANCED_SPATIAL_AUG_PARAMS: {
+          "enabled": False,
+          "scale1": 0.0,
+          "scale2": 0.0,
+          "stretch": 0.0,
+          "rotate": 0.0,
+          "translate": 0.0,
+          "enable_out_of_boundary_crop": False
+        }
+      VALIDATION:
+        SPATIAL_AUG_PARAMS: {"enabled": False}
+        COLOR_AUG_PARAMS: {"enabled": False}
+        ERASER_AUG_PARAMS: {"enabled": False}
+        FLIP_AUG_PARAMS: {"enabled": False}
+        ADVANCED_SPATIAL_AUG_PARAMS : {"enabled": False}
+OPTIMIZER:
+  NAME: AdamW
+  LR: 0.0004
+  PARAMS:
+    weight_decay: 0.0001
+    betas: [0.9, 0.999]
+    eps: 1.e-08
+    amsgrad: False
+GRAD_CLIP: 
+  USE: True
+  VALUE: 1.0
+TARGET_SCALE_FACTOR: 1.0
+MIXED_PRECISION: False
+DEVICE: "0"
+DISTRIBUTED:
+  USE: False
+  WORLD_SIZE: 2
+  BACKEND: nccl
+  MASTER_ADDR: localhost
+  MASTER_PORT: "12355"
+EPOCHS: null
+NUM_STEPS: null
+RESUME_TRAINING:
+  CONSOLIDATED_CKPT: null
+  EPOCHS: null
+  START_EPOCH: null
diff --git a/configs/trainers/_base_/kubric_improved_aug.yaml b/configs/trainers/_base_/kubric_improved_aug.yaml
@@ -0,0 +1,89 @@
+DATA:
+  TRAIN_DATASET:
+    NAME: "kubric"
+    ROOT_DIR: "./Datasets/KubricFlow"
+  VAL_DATASET:
+    NAME: "kubric"
+    ROOT_DIR: "./Datasets/KubricFlow" 
+  NUM_WORKERS: 4
+  PIN_MEMORY: True
+  APPEND_VALID_MASK: False
+  SHUFFLE: True
+  AUGMENTATION:
+    # Spatial Augmentation Settings borrowed from AutoFlow: https://github.com/google-research/opticalflow-autoflow/blob/main/src/dataset_lib/augmentations/aug_params.py
+    USE: True
+    PARAMS:
+      TRAINING:
+        COLOR_AUG_PARAMS: {
+          "enabled": True,
+          "asymmetric_color_aug_prob": 0.2, 
+          "brightness": 0.4, 
+          "contrast": 0.4, 
+          "saturation": 0.4, 
+          "hue": 0.15915494309189535
+        }
+        ERASER_AUG_PARAMS: {
+          "enabled": True,
+          "aug_prob": 0.5,
+          "bounds": [50, 100]
+        }
+        NOISE_AUG_PARAMS: {
+          "enabled": False,
+          "aug_prob": 0.5,
+          "noise_std_range": 0.06 
+        }
+        FLIP_AUG_PARAMS: {
+          "enabled": True, 
+          "h_flip_prob": 0.5, 
+          "v_flip_prob": 0.1
+        }
+        SPATIAL_AUG_PARAMS: {
+          "enabled": False,
+          "aug_prob": 0.0, 
+          "stretch_prob": 0.0, 
+          "min_scale": 0, 
+          "max_scale": 0, 
+          "max_stretch": 0
+        }
+        ADVANCED_SPATIAL_AUG_PARAMS: {
+          "enabled": True,
+          "scale1": 0.3,
+          "scale2": 0.1,
+          "rotate": 0.4,
+          "translate": 0.4,
+          "stretch": 0.3,
+          "enable_out_of_boundary_crop": False
+        }
+      VALIDATION:
+        SPATIAL_AUG_PARAMS: {"enabled": False}
+        COLOR_AUG_PARAMS: {"enabled": False}
+        ERASER_AUG_PARAMS: {"enabled": False}
+        FLIP_AUG_PARAMS: {"enabled": False}
+        ADVANCED_SPATIAL_AUG_PARAMS : {"enabled": False}
+OPTIMIZER:
+  NAME: AdamW
+  LR: 0.0004
+  PARAMS:
+    weight_decay: 0.0001
+    betas: [0.9, 0.999]
+    eps: 1.e-08
+    amsgrad: False
+GRAD_CLIP: 
+  USE: True
+  VALUE: 1.0
+FREEZE_BATCH_NORM: False
+TARGET_SCALE_FACTOR: 1.0
+MIXED_PRECISION: False
+DEVICE: "0"
+DISTRIBUTED:
+  USE: False
+  WORLD_SIZE: 2
+  BACKEND: nccl
+  MASTER_ADDR: localhost
+  MASTER_PORT: "12355"
+EPOCHS: null
+NUM_STEPS: null
+RESUME_TRAINING:
+  CONSOLIDATED_CKPT: null
+  EPOCHS: null
+  START_EPOCH: null