How to write good tests for parallel environments

[williamlardier]

What does this PR do, and why do we need it?

With 10 rules, ensure the new tests in CTST are following a set of rule to ensure their quality.

Which issue does this PR fix?

Fixes ZENKO-4944.

Special notes for your reviewers:

Please feel free to suggest changes or new rules, or start a discussion if you disagree with a rule.

[bert-e]

Hello williamlardier,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Incorrect fix version

The Fix Version/s in issue ZENKO-4944 contains:

• None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

• 2.10.12

Please check the Fix Version/s of ZENKO-4944, or the target
branch of this pull request.

[bert-e]

Incorrect fix version

The Fix Version/s in issue ZENKO-4944 contains:

• 2.10.11

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

• 2.10.12

Please check the Fix Version/s of ZENKO-4944, or the target
branch of this pull request.

[francoisferrand]

should this instead go in a more "natural" location, e.g. docs/writing-ctst-tests.md ?

[williamlardier]

The current docs folder is used with tox, it's a real doc we used to render as html pages, putting it here is logical but it doesn't look good...

[francoisferrand]

Suggested change

	> Tests must be idempotent.
	> Tests should be idempotent.

[williamlardier]

Why "should" and not "must"? it is not supposed to be a suggestion.

[francoisferrand]

Suggested change

	... must be created before the test start, or in a dedicated set of test.
	... should be created before the test start, or in a dedicated set of test.

[rachedbenmustapha]

General formatting/presentation remarks:

• It would be more straightforward for readers to just name the rules as titles prefixed by their number. I find the bold rule number followed by the rule itself as a quote a bit distracting.
• The word please is not needed in this sort of document (or any documentation or UI really) and sets a tone of asking for a favor vs giving directions that everyone needs to follow. A former tech writer on the team made sure that we removed all pleases from docs (even internal/technical) and UIs for consistency.

[rachedbenmustapha]

I both agree and disagree: tearing down resources should always be done and be reliable, and also we need to be able to debug. A middle ground would be to skip teardown on failure only if in debug mode (the one with tmate active), otherwise always do it.

This would make rule #9 redundant as tests relying on other tests to have run would just not work.

[williamlardier]

Thank you for all the feedback. I updated the page accordingly - I agree with everything but this. I updated as you suggested though.

I think we should not cleanup resources in case of error even if we're not running debug. The reason is that as each test uses a partitioned set of resources (bucket/objects, possibly account), not cleaning in case of error (or flaky) helps us finding the root cause much more easily, by having access to the metadata in the mongodb dumps, as well as the logs of all operations done by CTST for the failing test (identifiable with a single id). The overhead is minimal and it should not affect other tests in any way, even if the bucket is still here.