Reg cocktails #358

ravinkohli · 2021-12-08T17:45:54Z

Types of changes

Breaking change (fix or feature that would cause existing functionality to not work as expected)

Note that a Pull Request should only contain one of refactoring, new features or documentation changes.
Please separate these changes and send us individual PRs for each.
For more information on how to create a good pull request, please refer to The anatomy of a perfect pull request.

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.

Have you checked to ensure there aren't other open Pull Requests for the same update/change?
Have you added an explanation of what your changes do and why you'd like us to include them?
Have you written new tests for your core changes, as applicable?
Have you successfully ran tests with your changes locally?

Description

This PR contains code for the paper Regularization is all you need. The code is a duplicate of refactor_development_regularization_cocktails but it has been rebased with the development branch. It has not yet been benchmarked and compared to the refactor_development_regularization_cocktails, so there are no performance guarantees but it serves as the default branch for further works on Regularization Cocktails. It is not to be merged yet.

Motivation and Context

This PR is not to be merged right now.

How has this been tested?

I have added tests to ensure the basic functionality of the cocktails are working on top of the tests already present in development but they can and will definitely be improved.

codecov · 2021-12-10T12:33:09Z

Codecov Report

Base: 85.50% // Head: 85.16% // Decreases project coverage by -0.33% ⚠️

Coverage data is based on head (34c704d) compared to base (0e574af).
Patch coverage: 82.71% of modified lines in pull request are covered.

❗ Current head 34c704d differs from pull request most recent head 873df9a. Consider uploading reports for the commit 873df9a to get more accurate results

Additional details and impacted files

@@               Coverage Diff               @@
##           development     #358      +/-   ##
===============================================
- Coverage        85.50%   85.16%   -0.34%     
===============================================
  Files              231      240       +9     
  Lines            16303    17059     +756     
  Branches          3009     3155     +146     
===============================================
+ Hits             13940    14529     +589     
- Misses            1524     1653     +129     
- Partials           839      877      +38

Impacted Files	Coverage Δ
autoPyTorch/api/tabular_regression.py	`100.00% <ø> (ø)`
autoPyTorch/api/time_series_forecasting.py	`59.15% <0.00%> (-3.54%)`	⬇️
autoPyTorch/data/base_target_validator.py	`98.07% <ø> (ø)`
autoPyTorch/datasets/base_dataset.py	`80.14% <ø> (-0.14%)`	⬇️
autoPyTorch/datasets/resampling_strategy.py	`96.15% <ø> (ø)`
autoPyTorch/evaluation/tae.py	`69.84% <ø> (ø)`
autoPyTorch/evaluation/train_evaluator.py	`87.50% <ø> (ø)`
autoPyTorch/optimizer/smbo.py	`75.91% <ø> (-4.53%)`	⬇️
.../tabular_preprocessing/TabularColumnTransformer.py	`74.46% <ø> (ø)`
...nents/preprocessing/tabular_preprocessing/utils.py	`100.00% <ø> (ø)`
... and 91 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Fix mypy Initial implementation of adversarial training Modifying the code to have activation controlled batch normalization Adding activation controlled weight decay, updating the style for code style check Commit for passing style check Style check try 2 Bug fix Adding unit test for adversarial trainer Adding code for activation controlled skip connections, with the additional choice of shake-shake and shake-drop being hyperparameters as the choice for the multi branch networks Bug fix for the failing tests Adding better conditions Try at a fix Temporary fix for the failing test Failing code check Failing code check v2 Add new update to fix break Flake8 coding style fix Removing duplicate unit test In progress

swa working, se in progress Fixed bug in update model with swa model, add predict with snapshot ensemble; todo: add tests for both

Add pytest_mock to test dependencies lookahead in progress add lookahead hyperparameters

Fix flake mypy Fix error in test

Addressed comments from francisco fixed issues after merge Add displayable path to model file assertion Add test for non cyclic scheduler

Fix flake

add min for 1 Addressed comments from shuhei, better documentation

Added debug information for API Fix flake Fix import made test deterministic for feature preprocessing Fix bug in parsing log convert to int Fix bug in testing

* Updating search space * fix typo * Bug fix * Fixing buggy implementation of predict when using gpu bug fixes fixing code style checks bug fix for use_pynisher in the base pipeline bug fix

…s, removing use_pynisher since it is not used anymore, hack for the moment to bypass illegal hyperparameter ranges for conditions Fixing flake8 fail flake8 fix flake8 fix

Addressing the comment from Ravin Adding checks for the possible different value range of multibranch_choice Removing comment since it is not relevant anymore Removing unecessary code that transforms to cpu, while it is on cpu

* Add dropout shape as a hyperparameter * fix stupid bug

…ner (#214)

* Fix bugs in cutout training * Address comments from arlind

* Fixes for the development branch and regularization cocktails * Update implementation * Fix unit tests temporarily * Implementation update and bug fixes * Removing unecessary code * Addressing Ravin's comments [refactor] Address Shuhei's comments [refactor] Address Shuhei's comments [refactor] Address Shuhei's comments [refactor] Address Shuhei's comments

[fix] Fix Flake8 issues [refactor] Address Shuhei's comment [refactor] Address Shuhei's comments [refactor] Address Shuhei's comments [refactor] Address Shuhei's comments

* Update implementation * Coding style fixes * Implementation update * Style fix * Turn weighted loss into a constant again, implementation update * Cocktail branch inconsistencies (#275) * To nemo * Revert change in T_curr as results conclusively prove it should be 0 * Revert cutmix change after data from run * Final conclusion after results * FIX bug in shake alpha beta * Updated if is_training condition for shake drop * Remove temp fix in row cutmic * Cocktail fixes time debug (#286) * preprocess inside data validator * add time debug statements * Add fixes for categorical data * add fit_ensemble * add arlind fix for swa and se * fix bug in trainer choice fit * fix ensemble bug * Correct bug in cleanup * Cleanup for removing time debug statements * ablation for adversarial * shuffle false in dataloader * drop last false in dataloader * fix bug for validation set, and cutout and cutmix * shuffle = False * Shake Shake updates (#287) * To test locally * fix bug in trainer choice fit * fix ensemble bug * Correct bug in cleanup * To test locally * Cleanup for removing time debug statements * ablation for adversarial * shuffle false in dataloader * drop last false in dataloader * fix bug for validation set, and cutout and cutmix * To test locally * shuffle = False * To test locally * updates to search space * updates to search space * update branch with search space * undo search space update * fix bug in shake shake flag * limit to shake-even * restrict to even even * Add even even and others for shake-drop also * fix bug in passing alpha beta method * restrict to only even even * fix silly bug: * remove imputer and ordinal encoder for categorical transformer in feature validator * Address comments from shuhei * fix issues with ensemble fitting post hoc * Address comments on the PR * Fix flake and mypy errors * Address comments from PR #286 * fix bug in embedding * Update autoPyTorch/api/tabular_classification.py Co-authored-by: nabenabe0928 <[email protected]> * Update autoPyTorch/datasets/base_dataset.py Co-authored-by: nabenabe0928 <[email protected]> * Update autoPyTorch/datasets/base_dataset.py Co-authored-by: nabenabe0928 <[email protected]> * Update autoPyTorch/pipeline/components/training/trainer/base_trainer.py Co-authored-by: nabenabe0928 <[email protected]> * Address comments from shuhei * adress comments from shuhei * fix flake and mypy * Update autoPyTorch/pipeline/components/training/trainer/RowCutMixTrainer.py Co-authored-by: nabenabe0928 <[email protected]> * Update autoPyTorch/pipeline/tabular_classification.py Co-authored-by: nabenabe0928 <[email protected]> * Update autoPyTorch/pipeline/components/setup/network_backbone/utils.py Co-authored-by: nabenabe0928 <[email protected]> * Update autoPyTorch/pipeline/components/setup/network_backbone/utils.py Co-authored-by: nabenabe0928 <[email protected]> * Update autoPyTorch/pipeline/components/setup/network_backbone/utils.py Co-authored-by: nabenabe0928 <[email protected]> * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> * increase threads_per_worker * fix bug in rowcutmix * Enhancement for the tabular validator. (#291) * Initial try at an enhancement for the tabular validator * Adding a few type annotations * Fixing bugs in implementation * Adding wrongly deleted code part during rebase * Fix bug in _get_args * Fix bug in _get_args * Addressing Shuhei's comments * Address Shuhei's comments * Refactoring code * Refactoring code * Typos fix and additional comments * Replace nan in categoricals with simple imputer * Remove unused function * add comment * Update autoPyTorch/data/tabular_feature_validator.py Co-authored-by: nabenabe0928 <[email protected]> * Update autoPyTorch/data/tabular_feature_validator.py Co-authored-by: nabenabe0928 <[email protected]> * Adding unit test for only nall columns in the tabular feature categorical evaluator * fix bug in remove all nan columns * Bug fix for making tests run by arlind * fix flake errors in feature validator * made typing code uniform * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> * address comments from shuhei * address comments from shuhei (2) Co-authored-by: Ravin Kohli <[email protected]> Co-authored-by: Ravin Kohli <[email protected]> Co-authored-by: nabenabe0928 <[email protected]> * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> * resolve code issues with new versions * Address comments from shuhei * make run_traditional_ml function * implement suggestion from shuhei and fix bug in rowcutmixtrainer * fix return type docstring * add better documentation and fix bug in shake_drop_get_bl * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> * add test for comparator and other improvements based on PR comments * fix bug in test * [fix] Fix the condition in the raising error of all_nan_columns * [refactor] Unite name conventions of numpy array and pandas dataframe * [doc] Add the description about the tabular feature transformation * [doc] Add the description of the tabular feature transformation * address comments from arlind * address comments from arlind * change to as_tensor and address comments from arlind * correct description for functions in data module Co-authored-by: nabenabe0928 <[email protected]> Co-authored-by: Arlind Kadra <[email protected]> Co-authored-by: nabenabe0928 <[email protected]> * Addressing Shuhei's comments * flake8 problems fix * Update autoPyTorch/api/base_task.py Add indent. Co-authored-by: Ravin Kohli <[email protected]> * Update autoPyTorch/api/base_task.py Add indent. Co-authored-by: Ravin Kohli <[email protected]> * Update autoPyTorch/data/tabular_feature_validator.py Add indentation. Co-authored-by: Ravin Kohli <[email protected]> * Update autoPyTorch/pipeline/components/setup/network_backbone/utils.py Add line indentation. Co-authored-by: Ravin Kohli <[email protected]> * Update autoPyTorch/data/tabular_feature_validator.py Validate if there is a column transformer since for sparse matrices we will not have one. Co-authored-by: Ravin Kohli <[email protected]> * Update autoPyTorch/utils/implementations.py Delete uncommented line. Co-authored-by: Ravin Kohli <[email protected]> * Allow the number of threads to be given by the user * Removing unnecessary argument and refactoring the attribute. * Addressing Ravin's comments * Update autoPyTorch/pipeline/components/setup/network_backbone/utils.py Updating the function documentation according to the agreed style. Co-authored-by: Ravin Kohli <[email protected]> * Update autoPyTorch/pipeline/components/setup/network_backbone/utils.py Providing information on the wrong method provided for shake-shake regularization. Co-authored-by: nabenabe0928 <[email protected]> * add todo for backend and accept changes from shuhei * Addressing Shuhei's and Ravin's comments * Addressing Shuhei's and Ravin's comments, bug fix * Update autoPyTorch/pipeline/components/setup/network_backbone/ResNetBackbone.py Improving code readibility. Co-authored-by: nabenabe0928 <[email protected]> * Update autoPyTorch/pipeline/components/setup/network_backbone/ResNetBackbone.py Improving consistency. Co-authored-by: nabenabe0928 <[email protected]> * bug fix Co-authored-by: Ravin Kohli <[email protected]> Co-authored-by: nabenabe0928 <[email protected]> Co-authored-by: nabenabe0928 <[email protected]> Co-authored-by: Ravin Kohli <[email protected]>

* Initial fix for all tests passing locally py=3.8 * fix bug in tests * fix bug in test for data * debugging error in dummy forward pass * debug try -2 * catch runtime error in ci * catch runtime error in ci * add better debug test setup * debug some more * run this test only * remove sum backward * remove inplace in inception block * undo silly change * Enable all tests * fix flake * fix bug in test setup * remove anamoly detection * minor changes to comments * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> * Address comments from Shuhei * revert change leading to bug * fix flake * change comment position in feature validator * Add documentation for _is_datasets_consistent * address comments from arlind * case when all nans in test Co-authored-by: nabenabe0928 <[email protected]>

* update requirements * update requirements * resolve remaining conflicts and fix flake and mypy * Fix remaining tests and examples * fix failing checks * fix flake

* enable preprocessing and remove is_small_preprocess * address comments from shuhei and fix precommit checks * fix tests * fix precommit checks * add suggestions from shuhei for astype use * address speed issue when using object_dtype_mapping * make code more readable * improve documentation for base network embedding

* Enable learned embeddings, fix bug with non cyclic schedulers * add forbidden condition cyclic lr * refactor base_pipeline forbidden conditions * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> Co-authored-by: nabenabe0928 <[email protected]>

* fixed cut mix * remove unnecessary comment * change all_supported_metrics

* fixed cut mix * remove unnecessary comment * change all_supported_metrics * fix roc_auc for multiclass * remove unnecessary code

ravinkohli force-pushed the reg_cocktails branch from f45dbf2 to 87ee242 Compare December 8, 2021 17:55

ravinkohli marked this pull request as draft December 10, 2021 10:54

ravinkohli force-pushed the development branch from 00e2b96 to e3aeb55 Compare December 21, 2021 15:24

ravinkohli force-pushed the reg_cocktails branch from 9238a04 to 4b063f9 Compare December 21, 2021 17:01

ravinkohli force-pushed the reg_cocktails branch 2 times, most recently from 1c1ff8a to 2cdb40a Compare January 28, 2022 12:44

This was referenced Feb 10, 2022

Extra Dense Layer in the end of the network architecture #219

Open

Imbalanced dataset problems #195

Open

Dropout application #188

Open

[Addition] [Low Priority] Add more unit tests for the fixed search space implementation #179

Open

ravinkohli force-pushed the reg_cocktails branch from ea9cceb to 484ead4 Compare February 28, 2022 14:04

ravinkohli force-pushed the reg_cocktails branch from ce89f92 to 738be75 Compare March 9, 2022 17:33

ravinkohli force-pushed the development branch from 7f641b5 to 0e574af Compare July 18, 2022 18:28

franchuterivera and others added 16 commits July 21, 2022 12:12

Add cyclic property to lr scheduler and use_swa to trainer

3b4feae

swa working, se in progress Fixed bug in update model with swa model, add predict with snapshot ensemble; todo: add tests for both

ADD tests for SWA, SE

b845d4d

Add pytest_mock to test dependencies lookahead in progress add lookahead hyperparameters

ADD tests for LookAhead

8a2364d

Fix flake mypy Fix error in test

Added default for lookahead config

e2ee40a

Addressed comments from francisco fixed issues after merge Add displayable path to model file assertion Add test for non cyclic scheduler

Fix errors in Adversarial training

6e0d2bb

Fix pickling error for swa model

9290b07

Fix issues after rebase from refactor_development

c35c795

Fix flake

Added n_restarts as hyperparameter for CosineAnnealing

2512709

fix bug with ealy stopping and swa

41677fd

cont...

e236df0

add min for 1 Addressed comments from shuhei, better documentation

Addressed comments from arlind, change in T_mul and T_0 calculations

f29a7e0

Added debug information for API Fix flake Fix import made test deterministic for feature preprocessing Fix bug in parsing log convert to int Fix bug in testing

Updating search space (#156)

f141d06

* Updating search space * fix typo * Bug fix * Fixing buggy implementation of predict when using gpu bug fixes fixing code style checks bug fix for use_pynisher in the base pipeline bug fix

Adding constant clause in the testing module for hyperparameter range…

9339895

…s, removing use_pynisher since it is not used anymore, hack for the moment to bypass illegal hyperparameter ranges for conditions Fixing flake8 fail flake8 fix flake8 fix

Updating implementation for tabular regression

126f7d4

Addressing the comment from Ravin Adding checks for the possible different value range of multibranch_choice Removing comment since it is not relevant anymore Removing unecessary code that transforms to cpu, while it is on cpu

Fixing buggy implementation of the network head with constant updates

f6f05ba

ravinkohli and others added 20 commits July 21, 2022 12:20

Add dropout shape as a hyperparameter (#213)

bc0540b

* Add dropout shape as a hyperparameter * fix stupid bug

Change weighted loss to categorical and fix for test adversarial trai…

5d6062f

…ner (#214)

added no head (#218)

622c185

Fix bugs in cutout training (#233)

c4b7729

* Fix bugs in cutout training * Address comments from arlind

[refactor] Address Shuhei's comments

c1a73f8

[fix] Fix Flake8 issues [refactor] Address Shuhei's comment [refactor] Address Shuhei's comments [refactor] Address Shuhei's comments [refactor] Address Shuhei's comments

[doc] Add referencing to each regularization techniques

769e041

[fix] Address Ravin's comments and fix range issues in row cut

0da4f72

[doc] Add the reference to the fit_dictionary

c4a4565

[FIX] Tests after rebase of reg_cocktails (#359)

02e97a1

* update requirements * update requirements * resolve remaining conflicts and fix flake and mypy * Fix remaining tests and examples * fix failing checks * fix flake

rebase and fix flake

03ddb64

fix merge conflicts after rebase

59b5830

fixes after rebase

c1fffa1

fixes after rebase

637a68b

fix tests after rebase

e69ff3b

fix mypy and flake

c138173

ravinkohli force-pushed the reg_cocktails branch from 738be75 to c138173 Compare July 26, 2022 13:43

ravinkohli marked this pull request as ready for review July 26, 2022 13:44

ravinkohli marked this pull request as draft July 26, 2022 13:44

ravinkohli and others added 2 commits July 26, 2022 15:54

fix silly removal of lightgbm

afddca5

[add] documentation update in base trainer (#468)

34c704d

ravinkohli added the enhancement New feature or request label Aug 19, 2022

ravinkohli added 2 commits September 23, 2022 17:39

[FIX] apply cutout for each row. (#481)

d29d11b

* fixed cut mix * remove unnecessary comment * change all_supported_metrics

[FIX] ROC AUC for multi class classification (#482)

873df9a

* fixed cut mix * remove unnecessary comment * change all_supported_metrics * fix roc_auc for multiclass * remove unnecessary code

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reg cocktails #358

Reg cocktails #358

ravinkohli commented Dec 8, 2021 •

edited

Loading

codecov bot commented Dec 10, 2021 •

edited

Loading

Reg cocktails #358

Are you sure you want to change the base?

Reg cocktails #358

Conversation

ravinkohli commented Dec 8, 2021 • edited Loading

Types of changes

Checklist:

Description

Motivation and Context

How has this been tested?

codecov bot commented Dec 10, 2021 • edited Loading

Codecov Report

ravinkohli commented Dec 8, 2021 •

edited

Loading

codecov bot commented Dec 10, 2021 •

edited

Loading