Improve tests

[jonasraoni]

Problems

• The tests take too long to run
• Running the tests locally is painful
• If a test fails, it forces us to run from beginning (or adopt a database/file backup strategy)
• Creating data for the tests can be challenging
• Frontend tests are tightly dependent on the backend
• Unable to paralelize tests due to race conditions
• Unable to test specific features on the e2e tests (it's possible on PHPUnit)

[jonasraoni]

Ideas

• Installing the database on a memory drive/ramdisk should improve the performance (it's needed to check if it fits the GitHub limits).
• Tools to reset/seed the database: mass create default submissions, users, reviews, metrics, etc.
  • Aren't the APIs enough to create data? What are they missing? There are UI tests that create a bunch of users, this type of thing could be moved to API calls.
  • Dumping/restoring a seeded database is pretty slow, but resetting a dockerized database is fast.
  • It's also needed to run tests with an empty dataset (e.g. installation, empty lists, etc)
• Make the frontend tests more granular/less dependent on each other.
• A faker package might be helpful on the PHP side, as well as on the JavaScript side.
• The frontend can have a switch to make use of mocked data to speed up tests (good for local tests/development time) and use the backend when there's intention to merge (probably will lead to maintenance burden).
• Spinning off more database instances might be enough to paralelize the tests, but GitHub limits might be a blocker.

[asmecher]

I think a good (and nearly work-ready) first step would be to separate the data seeding from the integration testing.

For creating the test dataset (seeding), I suggest we follow Laravel's database seeding approach. This means a series of PHP scripts that work directly with the storage layer (Eloquent models) to create the necessary data.

This is a little better abstracted than e.g. SQL scripts, and will perform far better than Cypress tests. It doesn't require 100% API coverage, though that is a laudable goal. And it's consistent with good Laravel approaches, which we'd do well to align further with.

If the current test dataset could be fully generated that way, it would free up the Cypress toolset to do what it's best at, while still allowing us the very useful https://github.com/pkp/datasets repo. It would also be a ton faster and more scalable.