Methods to validate inference results for sequential estimates

[Dingel321]

Hello,

As there are various methods for validating amortized posterior estimates (such as SBC, local-C2ST, and posterior predictive check), I was wondering if there are any methods that also work for a sequential posterior estimate, besides posterior predictive checks.

[michaeldeistler]

Hi there,

that's a great questions! Verifying sequential inference is much more tricky than verifying amortized methods.

In principle, all sequential methods (apart from TSNPE) converge to the correct posterior for all observations. Therefore, they would be amendable to running SBC, expected coverage, etc. However, in practice, these diagnostics might fail for sequential methods simply because sequential methods are inaccurate for a most observations (but typically more accurate for a single observation).

So, what to do? I would recommend to run local-C2ST, which returns correctness checks for individual observations (it is local) and evaluate the accuracy of the sequential inference method only on the desired observation.

Hope this helps!
Michael

[Dingel321]

Hi,

Thank you for your response!

In the tutorial with the local-C2ST, an amortized posterior is used. Therefore, I assumed it only works with one. But it makes sense to me that the classifier, which discriminates the true and learned joint distribution, learns that the posterior is pretty bad elsewhere besides the one observation we trained on.

Thank you!