Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add upgrade tests to circleCI #9970

Open
wants to merge 71 commits into
base: main
Choose a base branch
from

Conversation

carolinaecalderon
Copy link
Contributor

@carolinaecalderon carolinaecalderon commented Sep 23, 2024

Ticket

CM-449

Description

Add a new suite of test-e2e-release-upgrade tests that:

  • start a devcluster on a past Determined version (i.e., 0.34.0, 0.35.0, 0.36.0, etc)
  • stop the cluster
  • upgrade the cluster to the latest commit SHA on that branch
  • start the cluster again
  • run a subset of the e2e tests

For this iteration, I chose to omit a subset of the "regular" e2e tests that use SLURM, aws, or gcp -- I can add these back in later, if desired.

Commentary

Please refer to https://hpe-aiatscale.atlassian.net/wiki/spaces/ENGINEERIN/pages/1716125737/Cookbook+Release+Party#4.-Upgrade-Tests for more context on how these tests are running.

Test Plan

Observe that CircleCI passes: https://app.circleci.com/pipelines/gh/determined-ai/determined?branch=carolinac%2Fupgrade-tests

Checklist

  • Changes have been manually QA'd
  • New features have been approved by the corresponding PM
  • User-facing API changes have the "User-facing API Change" label
  • Release notes have been added as a separate file under docs/release-notes/
    See Release Note for details.
  • Licenses have been included for new code which was copied and/or modified from any external code

@cla-bot cla-bot bot added the cla-signed label Sep 23, 2024
Copy link

netlify bot commented Sep 23, 2024

Deploy Preview for determined-ui canceled.

Name Link
🔨 Latest commit 06fd8db
🔍 Latest deploy log https://app.netlify.com/sites/determined-ui/deploys/6720ed506decc00008bca7c1

Copy link

codecov bot commented Oct 2, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 54.49%. Comparing base (0013fd0) to head (06fd8db).
Report is 43 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9970      +/-   ##
==========================================
+ Coverage   54.47%   54.49%   +0.02%     
==========================================
  Files        1267     1267              
  Lines      159437   159437              
  Branches     3637     3637              
==========================================
+ Hits        86856    86890      +34     
+ Misses      72448    72414      -34     
  Partials      133      133              
Flag Coverage Δ
backend 45.73% <ø> (+0.06%) ⬆️
harness 72.55% <ø> (ø)
web 54.02% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

see 7 files with indirect coverage changes

@carolinaecalderon carolinaecalderon changed the title Carolinac/upgrade tests feat: add upgrade tests to circleCI Oct 3, 2024
@determined-ci
Copy link
Collaborator

Docsite preview being generated for this PR.
You can (eventually) find the generated docsite here.

@determined-ci determined-ci added the documentation Improvements or additions to documentation label Oct 9, 2024
@carolinaecalderon carolinaecalderon changed the base branch from main to INFENG-382-release-redesign October 9, 2024 14:45
@determined-ci determined-ci removed the documentation Improvements or additions to documentation label Oct 9, 2024
Base automatically changed from INFENG-382-release-redesign to main October 18, 2024 17:32
@determined-ci
Copy link
Collaborator

Docsite preview being generated for this PR.
You can (eventually) find the generated docsite here.

1 similar comment
@determined-ci
Copy link
Collaborator

Docsite preview being generated for this PR.
You can (eventually) find the generated docsite here.

@determined-ci determined-ci added the documentation Improvements or additions to documentation label Oct 18, 2024
@determined-ci
Copy link
Collaborator

Docsite preview being generated for this PR.
You can (eventually) find the generated docsite here.

.circleci/real_config.yml Outdated Show resolved Hide resolved
.circleci/real_config.yml Outdated Show resolved Hide resolved
@@ -4721,6 +4762,243 @@ workflows:
requires:
- package-and-push-system-dev-ee

test-e2e-upgrade:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm not sure i understand the value in these tests. if i understand, each e2e run runs determined $OLD_VERSION, doesn't do anything on that version, stops it, upgrades to the $HEAD, and then runs a test suite. but i don't see much of a difference between that and just testing against $HEAD to begin with; the only difference between those systems would be that one took a brief pause at some DB migration number, but even then they all the same migrations in the same order with essentially the same data. if we created some data or an experiment on $OLD_VERSION then upgraded to $HEAD and made sure the data looked right or the experiment successfully unpaused, that would cover a lot more of our previously experienced upgrade bugs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants