multi-fidelity optimization - fixed fidelity value

[ToennisStef]

Hello and happy new year!

I am currently working with the linear multi-fidelity model. I specifically don't want to do multi-fidelity-optimization. I just want to use the low-fidelity observations to improve the predictive performance of the model in the highest fidelity. I then want to do Bayesian optimization using the high-fidelity prediction of the model.

I was able to do so, by fixing the "fidelity" variable to the highest-fidelity value during the optimization using "context variables".

But when calculating the expected improvement for the highest fidelity I realized that the acquisition function values were incredibly small, and the next observation point was not really influenced by the mean value of the prediction of the highest fidelity function but only on the variance. Therefore I was not able to find the global minimum even after 100 Bayesian optimization steps even though the optimization problem is very simple (specific multi-fidelity forrester function with 4 fidelity function (see: Mainini et al. 2022. Analytic Benchmark Problems for Multifidelity Optimization Methods). Additionally using a "standard" GP with only high-fidelity observations, I'm able to find the minimum in only approx 10-20 steps.

I realized that the best observed y value for the expected improvement is selected from the whole y training set (including y values of the high-fidelity observation AND y-values of the low-fidelity observation). In my case, I had low-fidelity observations that were far below the mean prediction of highest-fidelity. Therefore the next observation point in each BO step was pretty much only influenced by the variance of the high-fidelity observation. It is very simple to create a custom acquisition function so that when evaluating only the best observed y-value of the high-fidelity is considered.

But my general question therefore is: when evaluating the acquisition function for a specific fidelity shouldn't the best observed y-value be only selected from the specific fidelity?

Best regards,
Stefan