Use fixture to run different config on same tests

[TimoKramer]

Since this was originally part of the refactor-test-namespace-story I was thinking about this some more. Each test-namespace would have to be adapted to use multiple configs and run all the tests for each config.

[TimoKramer]

On how to run a test multiple times with different fixture

[kordano]

Yes, that's really something that we need. With all different configs and backends.

[jsmassa]

I needed all the tests to run for the persistent set when I was trying out the durable version, but most of them are only using the default config which sets the index structure to the hitchhiker-tree. For a while, I was simply changing the default value for my branch and - voila - a lot more bugs were caught. Then, wanting to do it properly, I thought of a framework to create all configs of interest and started to convert test by test to iterate through all of them. Fortunately, I didn't come very far because one night I had a different idea which I though was genius at the time and made me happy for at least a day xD

Long story short, my idea was to set the default config variables from outside, i.e. bind them in kaocha's tests.edn for different test setups. With that, we can run the tests for a particular setup locally and run all of the setups in the CI/CD pipeline.

I just saw this discussion now, otherwise I guess I wouldn't have looked for a different solution, but the dynamic default solution actually seems more flexible to me then the fixture solution. I wouldn't say I have too much experience with test fixtures though, so I wouldn't be surprised if they are more powerful than I think. Besides, I don't know if there is a downside to making the default values dynamic.

[TimoKramer]

I guess if you find a different approach it is probably better than fixtures.