Thoughts on Adding UnionHyperparameter

[thomasjpfan]

Currently the method to implement a "union" type is to use conditional hyperparameters:

import ConfigSpace as CS
from ConfigSpace.hyperparameters import CategoricalHyperparameter as CatHP
from ConfigSpace.hyperparameters import UniformFloatHyperparameter as UniFloatHP
from ConfigSpace.hyperparameters import UniformIntegerHyperparameter as UniIntHP
from ConfigSpace.conditions import EqualsCondition as EqualCond

cs = CS.ConfigurationSpace()
hp = {
    'min_samples_split:c': CatHP('max_features_c', ['int', 'float'], 
                                 default_value='int'),
    'min_samples_split:c:int':  UniIntHP('min_samples_leaf:control:int', 
                                         1, 20, default_value=1),
    'min_samples_split:c:float':  UniFloatHP('min_samples_split:control:float', 
                                             0.01, 1.0)
}

conditions = [
    EqualCond(hp['min_samples_split:c:int'], hp['min_samples_split:c'], 'int'),
    EqualCond(hp['min_samples_split:c:float'], hp['min_samples_split:c'], 'float'),
]

cs2.add_hyperparameters(hp.values())
cs2.add_conditions(conditions)

It would be interesting to wrap this logic into a UnionHyperparameter datatype:

cs2 = CS.ConfigurationSpace()
hp2 = UnionHyperparameter("union", hyperparams=[
     UniIntHP('min_samples_leaf_int', 1, 20, default_value=1),
     UniFloatHP('min_samples_leaf_float', 0.01, 1.0)
])

The most natural way of representing this would be a tuple of length 2, one to choose the hyperparams, and the second for the actual value. This may not work in ConfigSpace, because the code base assumes that the vector representation has one value per hyper-parameter.

Another way to do this is to map the choices in UnionHyperparameter to different sections of the real line.

@mfeurer What are your thoughts on a UnionHyperparameter?

[mfeurer]

Thanks for this suggestion. Indeed, having multiple types and ranges for single hyperparameter would be great to model spaces like the search space of scikit-learn. Can you think of any other use case of this feature?

Besides the issue you mentioned (ConfigSpace assumes that there's a one-to-one mapping between hyperparameters and the vector representation) it will also add a bit of burden on hyperparameter optimization packages using the ConfigSpace (okay, these are only two) because they need to implement additional logic to handle such union hyperparameters (vs. the hpo user currently having to implement this).

CC @mlindauer @KEggensperger @AndreBiedenkapp @frank-hutter