Handling of parameters with integer values

[MariusCautun]

Treating integer-valued parameters as floats can be sub-optimal for the maximization procedure. This is discussed in https://arxiv.org/abs/1706.03673, where the authors have shown that accounting for the integer nature of some parameters can lead to faster convergence (see for example their figure 3). Given that optimizing the machine learning hyperparameters involves many times integer parameters, I think it is a worthwhile problem.

The same paper proposes a simple way of dealing with integer parameters with minimal changes to the code. For backward compatibility, the simplest way to implement the changes would be to add an extra dictionary where you specify the integer variables (if no such dictionary is given, then all are assumed to be floats).

Is this a change worth considering? If so, I can give it a try and implement it as a pull request.

[bwheelz36]

Have you read this example, item 2:
https://github.com/fmfn/BayesianOptimization/blob/master/examples/advanced-tour.ipynb
?

If so, how would what you propose differ? (sorry, I haven't read the paper!)

[MariusCautun]

I saw the example in the advanced tour. The idea is that if you know a parameter is an integer then the Bayesian Optimizer can find the maximum faster (i.e achieve convergence in a smaller number of samples) than if you were to treat that parameter as a float.

The quickest would be to have an extra list or dictionary where you specify all the integer parameters. Then, when you calculate the Gaussian Process value for a point in parameter space, you round to integer values all the parameters that are defined as integers. The paper I pointed to claims that this extra step, which has minimal computation overhead, should speed up finding the maximum.

The idea is that if you have an integer parameter and you know the value of your black-box function at n and n+1, then you know the value of that function in the interval [n,n+1]. If you treat that parameter as a float, then even if you know the value of the function at n and n+1, your function still has the freedom to take a different value (within the dispersion of the fitted Gaussian Process) in the (n,n+1) interval. This means that you know less information than in the case of an integer parameter. This is why implementing integer parameters can speed up finding the maximum.

[bwheelz36]

Certainly sounds interesting. I was wondering how this works with the acquisition function and I think their figure 1 explains that very nicely. I guess you saw my reply on your other proposal that pull requests will likely take quite some time. But I think many people would be interested in the ability to better handle integers, especially if you can replicate some of these results showing aster convergence.

[sanoj2021]

I would be interested in an integer feature and fastened convergence. Greets

[till-m]

I've had a closer look at categorical/integer parameters and it shouldn't be too hard to implement. The only thing that is unclear to me is how the acquisition function will be maximized -- finding extrema of mixed value functions is a nontrivial problem. In Garrido-Merchan/Hernandez-Lobato there's no mention of a specific method used to find the maximum, as far as I can tell. If anyone has suggestions as to how to solve this sub-problem, I would be interested.

[bwheelz36]

@till-m - should we close this for now given the issue in 430?

[till-m]

Implemented in #531 and published as a beta release, version 3.0.0b1. If someone is working on problems using typed optimization, I would love some getting some feedback from beta testers. :) Documentation for the feature is available here (example usage) and here (API reference).