Sytest is slow and takes 4 minutes to run a single test

[MadLittleMods]

Sytest is slow and takes 4 minutes (pretty consistent 3m40s run-time) to run a single test which makes it very draining to figure out any problem. Is this expected?

Most of it is all in the setup part of it.

docker run --rm -it -e POSTGRES=1 -e MULTI_POSTGRES=1 -e WORKERS=1 \
    -v ~/Documents/github/element/synapse:/src:ro \
    -v ~/Documents/github/element/synapse/logs:/logs \
    -v ~/Documents/github/element/sytest:/sytest:ro matrixdotorg/sytest-synapse:buster tests/48admin.pl
    
... 4 minutes later ...
+ exit 1

Since the Sytest failure only occurs in the MULTI_POSTGRES, WORKERS variation, I can't just run ./run-tests.pl directly.

---

The test could be ported to Complement but this is orthogonal and I'm just trying to debug a broken test in CI. And the failure might be specific to the Sytest environment.

Complement is also slow but at least is the way forward.

[richvdh]

Hrm, indeed. For me it's a fairly consistent 1m40s, but that's still quite slow.

There's a couple of things going on here, both of them stemming from the fact that the matrixdotorg/sytest-synapse images don't contain an installed version of synapse, but rather the dependencies for synapse (or at least, are meant to contain those dependencies). The idea here was to make it easier to hack on synapse without building a new image each time (which is complement's approach).

The first thing is that, since Synapse 1.68, there is a Rust build step. This downloads a whole bunch of Rust libraries, so not only is this slow, it also breaks OFFLINE mode. I think the build process for matrixdotorg/sytest-synapse needs to run the rust build step (somehow), so that the rust dependencies can be cached.

The other thing that might be biting you is that the synapse run script (scripts/synapse_sytest.py) runs poetry, which will install the exact set of python dependencies that your synapse working copy wants. The matrixdotorg/sytest-synapse images are updated nightly to match the dependencies for synapse develop, but if your matrixdotorg/sytest-synapse image doesn't quite closely match your working copy, you could be installing a whole bunch of python dependencies. The output from the docker run should show if this is the case.

[MadLittleMods]

@richvdh Thanks for taking the time to reply and look into the aspects here!

The other thing that might be biting you is that the synapse run script (scripts/synapse_sytest.py) runs poetry, which will install the exact set of python dependencies that your synapse working copy wants. The matrixdotorg/sytest-synapse images are updated nightly to match the dependencies for synapse develop, but if your matrixdotorg/sytest-synapse image doesn't quite closely match your working copy, you could be installing a whole bunch of python dependencies. The output from the docker run should show if this is the case.be installing a whole bunch of python dependencies. The output from the docker run should show if this is the case.

I think this was definitely the case (was seeing poetry go). I guess I should build my own sytest-synapse image before I try to run Sytest (to get the deps aligned) 🤔

[richvdh]

I guess I should build my own sytest-synapse image before I try to run Sytest (to get the deps aligned)

either that, or merge develop into your branch (and docker pull matrixdotorg/sytest-synapse:buster) so that the deps approximately match up.