Minimizing predictive variance as part of loss function as per "Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance"

[ackbar03]

Hi,

I am trying to implement the paper here (Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance) in GPytorch.

The objective function to be minimized in the paper (equation 3) is

l_ssl = 1/n * l_likelihood + alpha/m * l_variance

where n, alpha, and m are weights and l_variance corresponds to predictive variance, i.e.

l_variance=sum(Variance(y_hat))

Can i confirm that for Variance(y_hat), we can directly use the predicted variance values from the distribution classes? i.e. we can have something like

f_preds_labelled = model(x_labelled)
f_preds_unlabelled = model(x_unlabelled)

loss = -mll(output, train_y) + f_preds_unlabelled.variance.sum() / m * alpha

Is the .variance attribute differentiable and will it backpropagate with the loss step function?

Also, if possible, can I just confirm that the GP layer type to be used for this DKL should be Exact GP and not approximate / stochastic variational GP? I'm quite certain it is just an exact DKL since I am aware of the difference with stochastic variational DKL but here seems to be quite a few variations in GPytorch so just want to be sure.

Thanks a lot!

[wjmaddox]

Is the .variance attribute differentiable and will it backpropagate with the loss step function?

Yes, it should be differentiable wrt both the gp's parameters and the inputs.

Also, if possible, can I just confirm that the GP layer type to be used for this DKL should be Exact GP and not approximate / stochastic variational GP?

Yes, I believe that they use exact inference from a skim of their paper.

[ackbar03]

Thanks Wesley for the prompt reply!

Could you also help me with a follow up question, since I see you've also helped work on the codebase before. I see for ExactGP models, it needs to be initialized with the train_inputs and train_targets functions.

My inputs are images however and are obtained in batches through a torch dataloader. It is more similar to the example that combines GP with DenseNet but the example uses the SVDKL. And this one does not require a train_input and train_target for initialization.

How should I go about this with an ExactGP model? Is it ok if I just pass a None value to train_inputs and train_targets? I took a brief look at the source code and it doesn't seem to use the values except in the fantasy_gp_model or something and I don't think thats really applicable to me.

Thanks again

[wjmaddox]

In that case, you actually do want to use a SVGP rather than the exact GP. You can't really do minibatching with the exact GP.

[KasperNiklas]

@wjmaddox Would you be so kind to elaborate why you can't do minibatching with the exact GP?

[wjmaddox]

In general, the gradients wrt the hypers will be incorrect. Although there's ways around that now (e.g. the approach in https://proceedings.neurips.cc/paper/2020/file/1cb524b5a3f3f82be4a7d954063c07e2-Paper.pdf).