Create e2e debug util file

[shaspitz]

This PR adds a util file that allows devs to easily debug individual e2e tests from VSCode's run test | debug test functionality. Previously you could only debug individual e2e tests through customizing a launch.json for VSCode. For me, this workflow is not desirable, and sometimes buggy/inconsistent.

Notes:

• The method of using a launch.json to debug e2e tests is still fully function if that's your preference
• These new tests are not ran when make test-short is called, since they exist in the /testutil/e2e directory
• The better version of this PR would be to refactor each e2e test to use native golang testing instead of the testify suite package. That way you could debug each test (click the debug button) from the same file that the test behavior is defined. But unfortunately, using the testify suite package is unavoidable if you wish to make your tests cleanly importable for external repos.

[shaspitz]

Tests are ordered as they appear in the VSCode explorer

[jtremback]

Convenient

[MSalopek]

Pretty neat! We should remember to add new e2e tests to util when we write them.

[danwt]

@smarshall-spitzbart could we try this approach instead?

golang/vscode-go#2414 (comment)

I tried it in a demo and it worked. It would let us avoid reflection and using hardcoded strings ect.

[shaspitz]

golang/vscode-go#2414 (comment)

Defining a new suite for each file would allow us to debug tests by file, but still not debug a single test on it's own. I have yet to find another way to debug a standalone e2e test

Also, our e2e tests cannot be defined in _test.go files since they need to be compiled and importable to other repos. This means the user would still have to start the debugging from a different file where the new suites are defined

[danwt]

Defining a new suite for each file would allow us to debug tests by file, but still not debug a single test on it's own. I have yet to find another way to debug a standalone e2e test

No this method definitely works! Diff tests use it and I tried it for e2e locally.

Also, our e2e tests cannot be defined in _test.go files since they need to be compiled and importable to other repos. This means the user would still have to start the debugging from a different file where the new suites are defined

Ah right that does seem like a limitation.

[shaspitz]

No this method definitely works! Diff tests use it and I tried it for e2e locally.

You still have to configure a JSON for your mentioned approach then? Otherwise I'm curious how you'd specify an individual test to debug. One of the main value props of this PR is avoiding JSON configuration

I agree that reflection is not always the best solution to things. But in this case, reflection enables a simple debug workflow: one click for debugging a single e2e test, no other configuration needed

On future repos like scheduler, settlement layer, etc., I'd doubt we'd need to make our e2e tests (or whatever we call them later on 😉) importable to external repos. So e2e tests could be written as native golang tests in _test.go files, keeping things strait forward and debug friendly

[danwt]

to configure a JSON

No, you've never had to configure a json. Even for diff tests. The json is just convenient IMO as it lets you debug using f5 without having to navigate to the test in the file or on the sidebar and clicking the green arrow. You can also specify e.g. command line args in the json if your program reads from stdin.

Put simply - you just go to the test and click the green arrow ;)

[shaspitz]

One suite per e2e file is not the same as one suite per e2e test, see slashing.go for example. It's not clear to me how your idea would allow me to run a single e2e test without touching a JSON file. There is not a green arrow that shows up for me when trying out the idea, since none of the e2e files are able to be test files

[danwt]

none of the e2e files are able to be test files

If you cant have any _test files then it indeed doesn't work. It works fine if _test files are allowed, without json.

[shaspitz]

Got it and makes sense. Yea that's the limiting factor here, we have to be able to export e2e tests in the compiled files