Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvement/zenko 4941 #2178

Open
wants to merge 11 commits into
base: development/2.6
Choose a base branch
from
Open

Conversation

williamlardier
Copy link
Contributor

@williamlardier williamlardier commented Nov 28, 2024

Complete tests of Quota feature in CTST.

-> custom images to be deleted

Another test to review starting the 2.8 branch:

development/2.8...w/2.8/improvement/ZENKO-4941#diff-0f9a8cc11d2560e9e9ea8668d53be732997ae7cecb4ec7201c836bc8367fc5c9R182

-> All tests are passing:

2024-11-29T09:56:42.1860871Z   @2.8.0 @PreMerge @Quotas @Restore @Dmf @ColdStorage @Only
2024-11-29T09:56:42.1861891Z   Scenario Outline: Restored object expiration updates quotas # features/quotas/Quotas.feature:182
2024-11-29T09:56:42.1880965Z     Given an action "RestoreObject"
2024-11-29T09:56:42.1883582Z     And a STORAGE_MANAGER type
2024-11-29T09:56:42.9398659Z     And a transition workflow to "e2e-cold" location
2024-11-29T09:56:47.9621673Z     And an upload size of 100 B for the object "obj-1"
2024-11-29T09:56:47.9946235Z     Then object "obj-1" should be "transitioned" and have the storage class "e2e-cold"
2024-11-29T09:56:57.1634848Z     Given a bucket quota set to 0 B
2024-11-29T09:56:57.1635502Z     And an account quota set to 0 B
2024-11-29T09:56:57.1638992Z     And a ACCOUNT type
2024-11-29T09:56:57.1639488Z     And an environment setup for the API
2024-11-29T09:56:57.1641680Z     And an "existing" IAM Policy that "applies" with "ALLOW" effect for the current API
2024-11-29T09:56:57.1654784Z     When i restore object "" for 5 days
2024-11-29T09:56:57.1823329Z     Then the API should "succeed" with ""
2024-11-29T09:56:57.1823860Z     And object "obj-1" should be "restored" and have the storage class "e2e-cold"
2024-11-29T09:57:04.2913305Z     Given a STORAGE_MANAGER type
2024-11-29T09:57:04.2913874Z     Then object "obj-1" should expire in 5 days
2024-11-29T09:57:04.3184634Z     When i wait for 5 days
2024-11-29T09:57:09.3217840Z     Then object "obj-1" should be "cold" and have the storage class "e2e-cold"
2024-11-29T09:57:21.6862371Z 
2024-11-29T09:57:21.6891444Z   @2.8.0 @PreMerge @Quotas @Restore @Dmf @ColdStorage @Only
2024-11-29T09:57:21.6892387Z   Scenario Outline: Restored object expiration updates quotas # features/quotas/Quotas.feature:182
2024-11-29T09:57:21.6903644Z     Given an action "RestoreObject"
2024-11-29T09:57:21.6915643Z     And a STORAGE_MANAGER type
2024-11-29T09:57:22.3246302Z     And a transition workflow to "e2e-cold" location
2024-11-29T09:57:27.3445935Z     And an upload size of 100 B for the object "obj-1"
2024-11-29T09:57:27.3756834Z     Then object "obj-1" should be "transitioned" and have the storage class "e2e-cold"
2024-11-29T09:57:37.5580469Z     Given a bucket quota set to 101 B
2024-11-29T09:57:37.5784356Z     And an account quota set to 101 B
2024-11-29T09:57:37.5973259Z     And a ACCOUNT type
2024-11-29T09:57:37.5973875Z     And an environment setup for the API
2024-11-29T09:57:37.5974634Z     And an "existing" IAM Policy that "applies" with "ALLOW" effect for the current API
2024-11-29T09:57:37.5978067Z     When i restore object "" for 5 days
2024-11-29T09:57:37.6263509Z     Then the API should "succeed" with ""
2024-11-29T09:57:37.6264026Z     And object "obj-1" should be "restored" and have the storage class "e2e-cold"
2024-11-29T09:57:44.7328078Z     Given a STORAGE_MANAGER type
2024-11-29T09:57:44.7329148Z     Then object "obj-1" should expire in 5 days
2024-11-29T09:57:44.7473779Z     When i wait for 5 days
2024-11-29T09:57:49.7499272Z     Then object "obj-1" should be "cold" and have the storage class "e2e-cold"
2024-11-29T09:51:13.8625920Z   @2.6.0 @PreMerge @Quotas @CronJob @DataDeletion @NonVersioned
2024-11-29T09:51:13.8627154Z   Scenario Outline: Quotas are affected by deletion operations between count items runs # features/quotas/Quotas.feature:107
2024-11-29T09:51:13.8628061Z     Given an action "DeleteObject"
2024-11-29T09:51:13.8628617Z     And a permission to perform the "PutObject" action
2024-11-29T09:51:13.8629197Z     And a STORAGE_MANAGER type
2024-11-29T09:51:14.2009069Z     And I PUT an object with size 100
2024-11-29T09:51:14.5443986Z     And a bucket quota set to 10000 B
2024-11-29T09:51:14.5599516Z     And an account quota set to 10000 B
2024-11-29T09:51:14.5727230Z     And an upload size of 1000 B for the object "obj-1"
2024-11-29T09:51:14.6027627Z     And a bucket quota set to 200 B
2024-11-29T09:51:14.6138121Z     And an account quota set to 200 B
2024-11-29T09:51:14.6227240Z     And a IAM_USER type
2024-11-29T09:51:14.6520382Z     And an environment setup for the API
2024-11-29T09:51:14.6521792Z     And an "existing" IAM Policy that "applies" with "ALLOW" effect for the current API
2024-11-29T09:51:14.6751194Z     When I wait 3 seconds
2024-11-29T09:51:14.8870100Z     When I wait 3 seconds
2024-11-29T09:51:14.9774911Z     Then the API should "fail" with "QuotaExceeded"
2024-11-29T09:51:14.9775549Z     When the "count-items" cronjobs completes without error
2024-11-29T09:51:15.1964713Z     When I wait 3 seconds
2024-11-29T09:51:17.6776293Z     And I PUT an object with size 100
2024-11-29T09:51:17.8890763Z     And i delete object "obj-1"
2024-11-29T09:51:17.9091224Z     And I wait 3 seconds
2024-11-29T09:51:18.1982800Z     And i delete object "obj-1"
2024-11-29T09:51:18.2191912Z     And I wait 3 seconds
2024-11-29T09:51:19.0025639Z     Then the API should "fail" with "QuotaExceeded"
2024-11-29T09:51:19.0026380Z     When the "count-items" cronjobs completes without error
2024-11-29T09:51:20.9103545Z     And I PUT an object with size 100
2024-11-29T09:51:20.9672979Z     Then the API should "succeed" with ""

@bert-e
Copy link
Contributor

bert-e commented Nov 28, 2024

Hello williamlardier,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval
/bypass_build_status Bypass the build and test status
/bypass_commit_size Bypass the check on the size of the changeset TBA
/bypass_incompatible_branch Bypass the check on the source branch prefix
/bypass_jira_check Bypass the Jira issue check
/bypass_peer_approval Bypass the pull request peers' approval
/bypass_leader_approval Bypass the pull request leaders' approval
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request TBA
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e
Copy link
Contributor

bert-e commented Nov 28, 2024

Incorrect fix version

The Fix Version/s in issue ZENKO-4941 contains:

  • None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 2.10.10

  • 2.6.72

  • 2.7.68

  • 2.8.48

  • 2.9.27

Please check the Fix Version/s of ZENKO-4941, or the target
branch of this pull request.

@williamlardier
Copy link
Contributor Author

/create_integration_branches

@scality scality deleted a comment from bert-e Nov 28, 2024
@williamlardier williamlardier force-pushed the improvement/ZENKO-4941 branch 2 times, most recently from 5e9e1d6 to 6841f9b Compare November 28, 2024 14:08
@scality scality deleted a comment from bert-e Nov 28, 2024
@bert-e
Copy link
Contributor

bert-e commented Nov 28, 2024

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

The following options are set: create_integration_branches

@scality scality deleted a comment from bert-e Nov 28, 2024
@williamlardier williamlardier force-pushed the improvement/ZENKO-4941 branch 2 times, most recently from 522a2b7 to 70d130c Compare November 28, 2024 16:06
@scality scality deleted a comment from bert-e Nov 28, 2024
@scality scality deleted a comment from bert-e Nov 28, 2024
@scality scality deleted a comment from bert-e Nov 28, 2024
@scality scality deleted a comment from bert-e Nov 28, 2024
@scality scality deleted a comment from bert-e Nov 28, 2024
@scality scality deleted a comment from bert-e Nov 28, 2024
@scality scality deleted a comment from bert-e Nov 28, 2024
@scality scality deleted a comment from bert-e Nov 28, 2024
@scality scality deleted a comment from bert-e Nov 28, 2024
@scality scality deleted a comment from bert-e Nov 28, 2024
@scality scality deleted a comment from bert-e Nov 28, 2024
@bert-e
Copy link
Contributor

bert-e commented Nov 28, 2024

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_integration_branches

@scality scality deleted a comment from bert-e Nov 28, 2024
@scality scality deleted a comment from bert-e Nov 28, 2024
@bert-e
Copy link
Contributor

bert-e commented Nov 28, 2024

Integration data created

I have created the integration data for the additional destination branches.

The following branches will NOT be impacted:

  • development/2.5

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

@bert-e
Copy link
Contributor

bert-e commented Nov 28, 2024

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

The following options are set: create_integration_branches

@williamlardier
Copy link
Contributor Author

ping

A race condition where a count items was running when the new
one started at the beggining of the quota tests, and the old
one completing after the new one, would lead to an override of
the infostore collection, in turn leading to missing metrics
for the quota buckets/accounts, making the quota tests fail
till a new count items starts.

Use a file locking mechanism to ensure no concurrent job runs

- Watching the resource or using kube api is prone to race
  condition, as their is a delay between the request to create
  a job, and its run, plus, the logic is not easy to centralize
  if we consider job already completed or not yet started, as
  we delete the old jobs upon completion
- Using alock mechanism is similar to what we already have for
  quotas, plus, it ensures no race condition and tests being
  able to restart as soon as possible.

Issue: ZENKO-4941
@scality scality deleted a comment from bert-e Dec 4, 2024
@bert-e
Copy link
Contributor

bert-e commented Dec 4, 2024

History mismatch

Merge commit #fbf25db87d2669b6b7877eaf2adf29f4c2075d0b on the integration branch
w/2.7/improvement/ZENKO-4941 is merging a branch which is neither the current
branch improvement/ZENKO-4941 nor the development branch
development/2.7.

It is likely due to a rebase of the branch improvement/ZENKO-4941 and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

The following options are set: create_integration_branches

@williamlardier
Copy link
Contributor Author

/force_reset

@bert-e
Copy link
Contributor

bert-e commented Dec 4, 2024

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_integration_branches

@scality scality deleted a comment from bert-e Dec 4, 2024
@scality scality deleted a comment from bert-e Dec 4, 2024
@bert-e
Copy link
Contributor

bert-e commented Dec 4, 2024

Integration data created

I have created the integration data for the additional destination branches.

The following branches will NOT be impacted:

  • development/2.5

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

@bert-e
Copy link
Contributor

bert-e commented Dec 4, 2024

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

The following options are set: create_integration_branches

@bert-e
Copy link
Contributor

bert-e commented Dec 4, 2024

Conflict

A conflict has been raised during the update of
integration branch w/2.8/improvement/ZENKO-4941 with contents from w/2.7/improvement/ZENKO-4941
and development/2.8.

Please resolve the conflict on the integration branch (w/2.8/improvement/ZENKO-4941).

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout w/2.8/improvement/ZENKO-4941
 $ git pull  # or "git reset --hard origin/w/2.8/improvement/ZENKO-4941"
 $ git merge origin/development/2.8
 $ # <intense conflict resolution>
 $ git commit
 $ git merge origin/w/2.7/improvement/ZENKO-4941
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.8/improvement/ZENKO-4941

The following options are set: create_integration_branches

- Ensure no scenario requiring count items to run at the same
  time, as they were overriding each other: if count items
  completes between 2 checks of a quota scenario, the "current"
  may be cleaned, leading to wrong test results.
- Incerase the count items lock stale duration: if we have
  too many buckets, we might fail to get the lock in time.
- Put the count items scenario in a dedicated folder, as it
  is not related to quotas.

Issue: ZENKO-4941
@bert-e
Copy link
Contributor

bert-e commented Dec 4, 2024

History mismatch

Merge commit #29a7622583ed2a0044029a19e3ce3b7e1065d667 on the integration branch
w/2.7/improvement/ZENKO-4941 is merging a branch which is neither the current
branch improvement/ZENKO-4941 nor the development branch
development/2.7.

It is likely due to a rebase of the branch improvement/ZENKO-4941 and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

The following options are set: create_integration_branches

@williamlardier
Copy link
Contributor Author

/force_reset

@bert-e
Copy link
Contributor

bert-e commented Dec 4, 2024

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_integration_branches

@scality scality deleted a comment from bert-e Dec 4, 2024
@scality scality deleted a comment from bert-e Dec 4, 2024
@bert-e
Copy link
Contributor

bert-e commented Dec 4, 2024

Integration data created

I have created the integration data for the additional destination branches.

The following branches will NOT be impacted:

  • development/2.5

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

@bert-e
Copy link
Contributor

bert-e commented Dec 4, 2024

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

The following options are set: create_integration_branches

- It may be prone to periodic instabilities
- Not sopping won't change the result and make the step more
  close to what the sentence says

Issue: ZENKO-4941
- We use 2s but the scuba frequency is 2s, so if we are unlucky,
  we may not be able to have it flushed, leading to flakies

Issue: ZENKO-4941
@bert-e
Copy link
Contributor

bert-e commented Dec 11, 2024

Incorrect fix version

The Fix Version/s in issue ZENKO-4941 contains:

  • 2.10.11

  • 2.6.73

  • 2.7.69

  • 2.8.49

  • 2.9.28

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 2.10.12

  • 2.6.74

  • 2.7.70

  • 2.8.50

  • 2.9.29

Please check the Fix Version/s of ZENKO-4941, or the target
branch of this pull request.

The following options are set: create_integration_branches


if (!fs.existsSync(lockFilePath)) {
fs.writeFileSync(lockFilePath, '');
}
Copy link
Contributor

@francoisferrand francoisferrand Dec 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the lock file exists, it means another test is waiting for this job : in that case, should the function not simply wait (polling) until the lock is removed? (i.e. the "first" test has completed its wait, and the job has indeed completed)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with polling like that is that two tests may detect that the file does not exist at the same time, hence the use of the lock file, that internally relies on directory creation (which is atomic) and last modified dates, to ensure the whole approach is safe.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of if (!fs.existsSync(lockFilePath)) { fs.writeFileSync(lockFilePath, '');, we can do fs.writeFileSync(path, data, { overwrite: false } : to atomically write the file without overwriting, so only one will ever succeed.
--> depending on the "outcome" of the creation, the code can decide if it should proceed with reading the cron/... or if it should simply wait because something else is already doing it.

Won't change the test, but the sentences are now more
meaningful.

Issue: ZENKO-4941
@@ -353,8 +353,7 @@ When('the user tries to perform the current S3 action on the bucket {int} times
}
await runActionAgainstBucket(this, this.getSaved<ActionPermissionsType>('currentAction').action);
if (this.getResult().err) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe better to keep failing on error, except for "retriable" errors : which can be safey ignore as this step is indeed about making multiple attempts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants