feat: add upgrade tests to circleCI

[carolinaecalderon]

Ticket

CM-449

Description

Add a new suite of test-e2e-release-upgrade tests that:

• start a devcluster on a past Determined version (i.e., 0.34.0, 0.35.0, 0.36.0, etc)
• stop the cluster
• upgrade the cluster to the latest commit SHA on that branch
• start the cluster again
• run a subset of the e2e tests

For this iteration, I chose to omit a subset of the "regular" e2e tests that use SLURM, aws, or gcp -- I can add these back in later, if desired.

Commentary

Please refer to https://hpe-aiatscale.atlassian.net/wiki/spaces/ENGINEERIN/pages/1716125737/Cookbook+Release+Party#4.-Upgrade-Tests for more context on how these tests are running.

Test Plan

Observe that CircleCI passes: https://app.circleci.com/pipelines/gh/determined-ai/determined?branch=carolinac%2Fupgrade-tests

Checklist

• Changes have been manually QA'd
• New features have been approved by the corresponding PM
• User-facing API changes have the "User-facing API Change" label
• Release notes have been added as a separate file under docs/release-notes/
  See Release Note for details.
• Licenses have been included for new code which was copied and/or modified from any external code

[netlify]

✅ Deploy Preview for determined-ui canceled.

Name	Link
🔨 Latest commit	06fd8db
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/6720ed506decc00008bca7c1

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 54.49%. Comparing base (0013fd0) to head (06fd8db).
Report is 43 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9970      +/-   ##
==========================================
+ Coverage   54.47%   54.49%   +0.02%     
==========================================
  Files        1267     1267              
  Lines      159437   159437              
  Branches     3637     3637              
==========================================
+ Hits        86856    86890      +34     
+ Misses      72448    72414      -34     
  Partials      133      133              

Flag	Coverage Δ
backend	45.73% <ø> (+0.06%)	⬆️
harness	72.55% <ø> (ø)
web	54.02% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

see 7 files with indirect coverage changes

[determined-ci]

Docsite preview being generated for this PR.
You can (eventually) find the generated docsite here.

[determined-ci]

Docsite preview being generated for this PR.
You can (eventually) find the generated docsite here.

[determined-ci]

Docsite preview being generated for this PR.
You can (eventually) find the generated docsite here.

[determined-ci]

Docsite preview being generated for this PR.
You can (eventually) find the generated docsite here.

[stoksc]

i'm not sure i understand the value in these tests. if i understand, each e2e run runs determined $OLD_VERSION, doesn't do anything on that version, stops it, upgrades to the $HEAD, and then runs a test suite. but i don't see much of a difference between that and just testing against $HEAD to begin with; the only difference between those systems would be that one took a brief pause at some DB migration number, but even then they all the same migrations in the same order with essentially the same data. if we created some data or an experiment on $OLD_VERSION then upgraded to $HEAD and made sure the data looked right or the experiment successfully unpaused, that would cover a lot more of our previously experienced upgrade bugs.