Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Increase to large instance size for scale tests #6184

Merged
merged 1 commit into from
May 15, 2024
Merged

fix: Increase to large instance size for scale tests #6184

merged 1 commit into from
May 15, 2024

Conversation

engedaam
Copy link
Contributor

@engedaam engedaam commented May 12, 2024 •
edited
Loading

Fixes #N/A

Description
1.) We are finding that nodes launched on with c6g.medium can't fit all the nodes provisioned by karpenter when using Bottlerocket AMIs. When we receive these instance type, we the scheduler not placing all the expected pods on those nodes. #5161. CW daemonset was added that that totaled to 158Mi.

2.) EC2NodeClass were not getting cleaned up as part of AfterSuite
3.) Karpenter was also getting CPU throttled at 3 CPUs, as the request and limits are set to 3 CPU. This is due to the increases in the number of pods from daemonSets
Screenshot 2024-05-12 at 3 31 15 PM

4.) Prometheus metrics collection prometheus-prometheus-kube-prometheus-prometheus-0 pods was seeing memory throttling with OOMKilled errors
Screenshot 2024-05-12 at 4 28 50 PM

How was this change tested?

Does this change impact docs?

  • Yes, PR includes docs updates
  • Yes, issue opened: #
  • No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

engedaam
engedaam commented May 12, 2024
View reviewed changes
Copy link
Contributor Author

@engedaam engedaam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/karpenter scale

@netlify Netlify
Copy link

netlify bot commented May 12, 2024 •
edited
Loading

✅ Deploy Preview for karpenter-docs-prod canceled.

Name Link
🔨 Latest commit 311e65d
🔍 Latest deploy log https://app.netlify.com/sites/karpenter-docs-prod/deploys/6643fc5cd47cde0008c8a2a7

engedaam
engedaam commented May 12, 2024
View reviewed changes
Copy link
Contributor Author

@engedaam engedaam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/karpenter scale

@coveralls
Copy link

coveralls commented May 12, 2024 •
edited
Loading

Pull Request Test Coverage Report for Build 9071477901

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 82.247%
Totals Coverage Status
Change from base Build 9071464346: 0.0%
Covered Lines: 5439
Relevant Lines: 6613
💛 - Coveralls

@github-actions GitHub Actions
Copy link
Contributor

Snapshot successfully published to oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter:0-ae0bf72d92239b6d1105564024d59ac5c52ccc71.
To install you must login to the ECR repo with an AWS account:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

helm upgrade --install karpenter oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter --version "0-ae0bf72d92239b6d1105564024d59ac5c52ccc71" --namespace "kube-system" --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

@github-actions GitHub Actions
Copy link
Contributor

Snapshot successfully published to oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter:0-56ee8a797b8d198647508481a4b3bd3a6a86ed88.
To install you must login to the ECR repo with an AWS account:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

helm upgrade --install karpenter oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter --version "0-56ee8a797b8d198647508481a4b3bd3a6a86ed88" --namespace "kube-system" --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

engedaam
engedaam commented May 12, 2024
View reviewed changes
Copy link
Contributor Author

@engedaam engedaam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/karpenter scale

engedaam
engedaam commented May 12, 2024
View reviewed changes
Copy link
Contributor Author

@engedaam engedaam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/karpenter scale

engedaam
engedaam commented May 12, 2024
View reviewed changes
Copy link
Contributor Author

@engedaam engedaam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/karpenter scale

@github-actions GitHub Actions
Copy link
Contributor

Snapshot successfully published to oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter:0-57f81c2e6750509b6f832d0adb2dd53e326e36f5.
To install you must login to the ECR repo with an AWS account:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

helm upgrade --install karpenter oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter --version "0-57f81c2e6750509b6f832d0adb2dd53e326e36f5" --namespace "kube-system" --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

engedaam
engedaam commented May 12, 2024
View reviewed changes
Copy link
Contributor Author

@engedaam engedaam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/karpenter scale

@github-actions GitHub Actions
Copy link
Contributor

Snapshot successfully published to oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter:0-de7542e7f4040f79d6ce0049d890b7376f174343.
To install you must login to the ECR repo with an AWS account:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

helm upgrade --install karpenter oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter --version "0-de7542e7f4040f79d6ce0049d890b7376f174343" --namespace "kube-system" --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

engedaam
engedaam commented May 12, 2024
View reviewed changes
Copy link
Contributor Author

@engedaam engedaam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/karpenter scale

engedaam
engedaam commented May 12, 2024
View reviewed changes
Copy link
Contributor Author

@engedaam engedaam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/karpenter scale

@github-actions GitHub Actions
Copy link
Contributor

Snapshot successfully published to oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter:0-060e6e0a065d772bec7f8934686c4ecb6ab32f96.
To install you must login to the ECR repo with an AWS account:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

helm upgrade --install karpenter oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter --version "0-060e6e0a065d772bec7f8934686c4ecb6ab32f96" --namespace "kube-system" --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

engedaam
engedaam commented May 13, 2024
View reviewed changes
Copy link
Contributor Author

@engedaam engedaam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/karpenter scale

@github-actions GitHub Actions
Copy link
Contributor

Snapshot successfully published to oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter:0-f1378f7a4da1ac8fc51831820a503843b186d219.
To install you must login to the ECR repo with an AWS account:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

helm upgrade --install karpenter oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter --version "0-f1378f7a4da1ac8fc51831820a503843b186d219" --namespace "kube-system" --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

engedaam
engedaam commented May 13, 2024
View reviewed changes
Copy link
Contributor Author

@engedaam engedaam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/karpenter scale

@engedaam engedaam marked this pull request as ready for review May 13, 2024 13:21
@engedaam engedaam requested a review from a team as a code owner May 13, 2024 13:21
@engedaam engedaam requested a review from tzneal May 13, 2024 13:21
engedaam
engedaam commented May 14, 2024
View reviewed changes
Copy link
Contributor Author

@engedaam engedaam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/karpenter scale

@github-actions GitHub Actions
Copy link
Contributor

Snapshot successfully published to oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter:0-0b8a7660886405919eed3d9baa5560ed8e6f39e4.
To install you must login to the ECR repo with an AWS account:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

helm upgrade --install karpenter oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter --version "0-0b8a7660886405919eed3d9baa5560ed8e6f39e4" --namespace "kube-system" --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

jmdeal
jmdeal approved these changes May 15, 2024
View reviewed changes
Copy link
Contributor

@jmdeal jmdeal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

@engedaam engedaam merged commit c93c945 into aws:main May 15, 2024
17 checks passed
@engedaam engedaam deleted the fix-scale-tests branch May 15, 2024 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jmdeal jmdeal jmdeal approved these changes

@tzneal tzneal Awaiting requested review from tzneal tzneal is a code owner automatically assigned from aws/karpenter

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@engedaam @coveralls @jmdeal