Unbound growth of .cache/go-build when running gremlins

[hknutzen]

I run gremlins on my project to check the tests of this project.
The project has more than 3000 test cases, all of them integration tests.
When running gremlins, it applies more than 5000 mutations.
I run gremlins with option --integration, so all tests are run for each mutation.
(The tests run with t.Parallel() and therefore complete in 1.1 seconds)

In recent days I started gremlins multiple times, playing with different versions and different options.
Yesterday, during a run of gremlins, there was no space left on the home partition of my development server.
It took some time until I found the reason:
The Go build cache in $HOME/.cache/go-build has grown to a size of 135 GBytes.
That was surprising.

I cleaned the cache and startet another run of gremlins.
It completed successfully:

Mutation testing completed in 2 hours 16 minutes
Killed: 3813, Lived: 416, Not covered: 23
Timed out: 104, Not viable: 871
Test efficacy: 90.16%
Mutator coverage: 99.46%

But the Go build cache has grown again:

du -hs .cache/go-build/
33G     .cache/go-build/

Gremlins compiles the project from different temporary directories again and again.
For each compiler run, Go fills the build cache, expecting, that the same directory gets compiled again.
But this never happens.

I could always clean the build cache manually.
But there seems to be an opportunity for optimization:

1. Either don't fill the build cache, as we know it will never be used.
2. Or reuse the same temporary directory for subsequent compiler runs,
   undoing the last mutation and applying the next mutation.
   I would expect a massive compile time improvement, since the Go build cache would be used now.

[k3rn31]

Hi @hknutzen, thanks for this report.

I see the problem and the possible optimisation. Actually, Gremlins already recycles the working directories as you describe. It creates one working directory per worker and each worker reuses its directory rolling back the mutations. You can see that the Dealer caches the worker per worker.
In the current version of Gremlins, in normal mode, the tests are cached by Go, but the cache is never used because the packages are continuously mutated.
In integration mode you can benefit from cacheing, though!

In the new timeout management I'm working on, I'm experimenting with cache disabling, but I'm still not convinced we have to disable it in integration mode, since some packages tests can be cached.

Maybe, one solution could be to clean the test cache at the end of the run with go clean -testcache, since we are sure the folders will change in the next run. But, is it correct to clean the whole cache for the user?
Maybe we can write a routine that cleans just the caches related to Gremlins runs...

[hknutzen]

I tried a go clean -testcache. But the size of the build cache remained unchanged at 33G.

[hknutzen]

When I run go test manually on my project, after cleaning the build cache, the new size is 26M.
I did some manual changes to the source code and executed go test again.
Each time, the size increased by ~ 9.5M.

Now lets calculate, which size we expect from a run of
gremlins run --coverpkg ./... -i --timeout-coefficient 24 --invert-logical --invert-loopctrl --invert-assignments
I use a machine with 12 cores.
Together with -i I expect 6 workers.
Gremlins applies 5227 mutations.
So I would expext this increase of the build cache:
6x26M + 5227x9,5M = 49812.5M ~= 50G

OK, then 33G isn't that bad.

[hknutzen]

I think gremlins should use a separate temporary directory as Go build cache.
This should be removed after each run of gremlins.
The users default cache wouldn't be affected any more

https://pkg.go.dev/cmd/go#hdr-Build_and_test_caching

The go command caches build outputs for reuse in future builds. The default location for cache data is a subdirectory named go-build in the standard user cache directory for the current operating system. Setting the GOCACHE environment variable overrides this default, and running 'go env GOCACHE' prints the current cache directory.

[k3rn31]

Great idea! Definitely we should use a separate cache folder.

I will change this issue into a discussion and I'll open an issue to address this change in gremlins.

[k3rn31]

I created issue #177 with high priority, I think we should address this as soon as possible.