Extend bucket notification timeout and map to the test timeout #2135

williamlardier · 2024-08-28T14:24:52Z

Increase timeout for bucket notification tests, to match the current config (ensure the check timeouts before the step)
Fixes an issue with kafka cleaner, when an error happens, we have no log and we clear the custer timeout, while the step timeout is -1, resulting in builds never completing (6h+), and improve the timeout.
Add a custom timeout when checking the DMF volume, as the default 100s is too low, and we have this issue very frequently since months.

bert-e · 2024-08-28T14:24:55Z

Hello williamlardier,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2024-08-28T14:25:02Z

Incorrect fix version

The Fix Version/s in issue ZENKO-4806 contains:

None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

2.10.0
2.6.66
2.7.62
2.8.42
2.9.17

Please check the Fix Version/s of ZENKO-4806, or the target
branch of this pull request.

Issue: ZENKO-4806

bert-e · 2024-09-02T13:45:09Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

williamlardier · 2024-09-02T13:45:19Z

/create_integration_branches

bert-e · 2024-09-02T13:45:30Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/ZENKO-4806 into
development/2.6
w/2.7/bugfix/ZENKO-4806 will be merged into development/2.7
w/2.8/bugfix/ZENKO-4806 will be merged into development/2.8
w/2.9/bugfix/ZENKO-4806 will be merged into development/2.9
w/2.10/bugfix/ZENKO-4806 will be merged into development/2.10

The following branches will NOT be impacted:

development/2.5

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2024-09-02T13:45:33Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

An issue where the kafkacleaner never completed was observed in the CI; as a result, the build took 6h before being cancelled. THis fix ensures that we tolerate up to 6min. Note that the current value is 30s * (1000 + 5000) so 3 minutes. Issue: ZENKO-4806

In some cases, the platform will take some time to process the event, so the test will fail, although there is no error. This change also add a sleep between two checks, to reduce the impact on the CPU when multiple tests are running this step in parallel. Issue: ZENKO-4806

Issue: ZENKO-4806

francoisferrand · 2024-09-03T10:38:45Z

tests/ctst/common/common.ts

@@ -237,14 +237,15 @@ Then('i {string} be able to add user metadata to object {string}',
        }
    });

-Then('kafka consumed messages should not take too much place on disk', { timeout: -1 },
+const kafkaCleanerTimeout = 1800000; // 30min


30 minutes seems verrrry long for kafka cleaner, which is setup to run every 30 second (or maybe 1-2 minutes) in CI...

It's already 30min today, this was the config where we got the best results

I am trying this now 0125893

francoisferrand · 2024-09-03T10:39:40Z

tests/ctst/common/common.ts

@@ -307,6 +308,9 @@ Then('kafka consumed messages should not take too much place on disk', { timeout

            // If a topic remains in this array, it means it has not been cleaned
            assert(topics.length === 0, `Topics ${topics.join(', ')} still have not been cleaned`);
+        } catch (err: unknown) {


is this not done in cucumber/ctst?
should it not be?

We have a "finally" that clears the timeout even if we got an error, which mean we never stop the test as its timeout was -1

Example: https://github.com/scality/Zenko/actions/runs/10668356308

what I mean is that (even with the catch) the exception should be propagated: and eventually reach cucumber/ctst where it shoudl be caught.... and immediately fail the test (even with no timeout)

Maybe the timeout was not due to an error... At least this ensures we log the error with a message

Errors not catched by the tests are sent to cucumber directly, it'll mark the test as failed. But in the build above, it didn't work

Issue: ZENKO-4806

williamlardier · 2024-10-03T14:02:48Z

/force_reset

bert-e · 2024-10-03T14:02:59Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_integration_branches

bert-e · 2024-10-03T14:03:11Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/ZENKO-4806 into
development/2.6
w/2.7/bugfix/ZENKO-4806 will be merged into development/2.7
w/2.8/bugfix/ZENKO-4806 will be merged into development/2.8
w/2.9/bugfix/ZENKO-4806 will be merged into development/2.9
w/2.10/bugfix/ZENKO-4806 will be merged into development/2.10

The following branches will NOT be impacted:

development/2.5

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2024-10-03T14:03:12Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

williamlardier · 2024-10-03T14:03:34Z

Last try to reproduce the errors with fresh branches, if I cannot, I will close this PR, as the current changes are now already part of other PRs

Extend bucket notification timeout and map to the test timeout

a2ec22a

Issue: ZENKO-4806

williamlardier force-pushed the bugfix/ZENKO-4806 branch from 9cecc0c to a2ec22a Compare September 2, 2024 13:44

scality deleted a comment from bert-e Sep 2, 2024

williamlardier added 3 commits September 3, 2024 11:38

Lint code

e22a9de

Issue: ZENKO-4806

francoisferrand reviewed Sep 3, 2024

View reviewed changes

Increase frequency and checks of kafkacleaner

0125893

Issue: ZENKO-4806

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extend bucket notification timeout and map to the test timeout #2135

Extend bucket notification timeout and map to the test timeout #2135

williamlardier commented Aug 28, 2024 •

edited

Loading

bert-e commented Aug 28, 2024

bert-e commented Aug 28, 2024

bert-e commented Sep 2, 2024

williamlardier commented Sep 2, 2024

bert-e commented Sep 2, 2024

bert-e commented Sep 2, 2024

francoisferrand Sep 3, 2024

williamlardier Sep 3, 2024

williamlardier Sep 3, 2024

francoisferrand Sep 3, 2024

williamlardier Sep 3, 2024 •

edited

Loading

williamlardier Sep 3, 2024

francoisferrand Sep 3, 2024

williamlardier Sep 3, 2024

williamlardier commented Oct 3, 2024

bert-e commented Oct 3, 2024

bert-e commented Oct 3, 2024

bert-e commented Oct 3, 2024

williamlardier commented Oct 3, 2024

Extend bucket notification timeout and map to the test timeout #2135

Are you sure you want to change the base?

Extend bucket notification timeout and map to the test timeout #2135

Conversation

williamlardier commented Aug 28, 2024 • edited Loading

bert-e commented Aug 28, 2024

Hello williamlardier,

bert-e commented Aug 28, 2024

Incorrect fix version

bert-e commented Sep 2, 2024

Request integration branches

williamlardier commented Sep 2, 2024

bert-e commented Sep 2, 2024

Integration data created

bert-e commented Sep 2, 2024

Waiting for approval

francoisferrand Sep 3, 2024

Choose a reason for hiding this comment

williamlardier Sep 3, 2024

Choose a reason for hiding this comment

williamlardier Sep 3, 2024

Choose a reason for hiding this comment

francoisferrand Sep 3, 2024

Choose a reason for hiding this comment

williamlardier Sep 3, 2024 • edited Loading

Choose a reason for hiding this comment

williamlardier Sep 3, 2024

Choose a reason for hiding this comment

francoisferrand Sep 3, 2024

Choose a reason for hiding this comment

williamlardier Sep 3, 2024

Choose a reason for hiding this comment

williamlardier commented Oct 3, 2024

bert-e commented Oct 3, 2024

Reset complete

bert-e commented Oct 3, 2024

Integration data created

bert-e commented Oct 3, 2024

Waiting for approval

williamlardier commented Oct 3, 2024

williamlardier commented Aug 28, 2024 •

edited

Loading

williamlardier Sep 3, 2024 •

edited

Loading