models with non-static outputs

[esgomezm]

Hi there,

Some GAN models (and probably others) do not always generate the same output so it's difficult to test that the output provided by the developer is the same as the one produced with the model.

We could fix a seed in the CI. However, this wouldn't ensure that the seed is the one set by the developer when creating the model and that it will work the same as in github actions.

You can find an example of such a model in this PR that I just corrected and is working even with deepImageJ: https://drive.google.com/file/d/1ujdaAZwXFRYofxyoDrpX0kD6HgsO1TM9/view?usp=sharing

Also, how much time do you think it can take us to correct this or accept the model in the zoo? (The main issue is that the model is perfectly working, and associated to a publication, so it would be nice to have it)

[constantinpape]

It sounds like we should only check that the model runs and output dimensions match for such a model.

[esgomezm]

yep, that sounds good. For this though, there should be some flag or info in the spec for the CI at least.
Also, we could provide the average difference between the test output and the computed one and give it as a warning in the model card

[FynnBe]

I don't understand why seeding has no hope of working?
I'd suggest to use seed 0 per convention, update our testing scripts such that we always set python, numpy and pytorch/tf seed 0 and then expect the model to pass... There might be other sources of randomness (see https://pytorch.org/docs/stable/notes/randomness.html), but we should at least try to have reproducible restults...

[esgomezm]

Ok, so then you suggest to fix all the seeds when creating the model (I will do it with the bioimageio library) and then, how do you know the seed when testing it? I'm happy to try if setting a seed works.
Also, if I'm running the bioimageio terminal command is different than being inside python and setting a seed. Any recommendation?

[FynnBe]

I would suggest to simply always use seed 0.
It wouldn't be set when predicting, but when testing we could always set it.

maybe test it manually and if that works I can add it to core to always set the seeds to 0 when testing

[esgomezm]

ok! I'll get back with something :)