enh: Post-Improving Boosting Models #1315
Labels
architecture
(re)design of existing or new framework subsystem
composer
Related to GP-composition algorithm
enhancement
New feature or request
Summary
TLDR:
Motivation
The motivation for refactoring boosting models was (#1155 #1209 #1264):
Results of testing on OpenML are available here. During the development, subsequent ideas for improvement arose.
Guide-level explanation
1. Updating initial assumptions with boosting models
Add more pipelines that contain boosting models to the initial assumptions. Updates presets by using boosting models. I want to draw your attention to the fact that boosting models have several strategies that can be used for various pipelines and presets. You can find more information about the strategy in the boosting framework documentation.
2. Evolutionary Mutations for Boosting Models
According to parameter updates, adding new mutations for pipelines with boosting models became possible.
Boosting strategy mutation:
Using category mutation:
enable_categorical
(default: True
)Change early_stopping_rounds:
early_stopping_rounds
to change the fitting time in the populationImproving metric of XGBoost model mutation:
max_depth
min_child_weight
,gamma
,lambda
Improving robustness for the noise of XGBoost model mutation:
subsample
,colsample_bytree
,colsample_bylevel
,colsample_bynode
for some step.dart
strategy method.Improving metric of LightGBM model mutation:
learning_rate
max_bin
,num_iterations
,num_leaves
dart
strategy method.Improving robustness for overfitting of LightGBM model mutation:
max_bin
andnum_leaves
min_data_in_leaf
andmin_sum_hessian_in_leaf
bagging_fraction
andbagging_freq
feature_fraction
lambda_l1
,lambda_l2
,min_gain_to_split
andextra_trees
max_depth
Improving robustness for overfitting of Catboost model mutation:
l2_leaf_reg
,colsample_bylevel
,subsample
max_depth
iterations
3. Allow the use of boosting model data with nan's.
One of the advantages of boosting methods is comparing with nan's in data. Implementing this feature in the current version requires refactoring the preprocessing with filling.
4. Allow to use boosting with GPU.
It is possible to use a GPU to accelerate fitting boosting. Therefore, it would be great to add such an opportunity.
Unresolved Questions
Is it possible to continue #1005? The main idea was to develop a method used in other AutoML frameworks. The main problem that had to be faced and not solved is that the pipeline generation approach differs from other frameworks, parallelization of the learning process of basic models on extraped samples, and embedding such an approach into the composition process since it is pretty time-consuming and resource-intensive. However, it guarantees more stable and accurate models. It would be possible to use this approach after composting; for example, if a boosting model is found in the final pipeline, then try to train it using this method.
P.S.
I also note that adding a weighted model, models from a family based on k-nearest neighbors as a meta-model in an ensemble, will help diversify pipelines for classification and regression.
Also note about not perfect method to detect categorical features.
The text was updated successfully, but these errors were encountered: