Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add upgrade tests to circleCI #9970

Open
wants to merge 71 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 69 commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
7495b03
feat: add support for upgrade tests to circleCI
carolinaecalderon Sep 23, 2024
97bb470
adding txt file; todo check branch build, add release-rc-filters, rem…
carolinaecalderon Oct 9, 2024
6be4f25
pushing & packing commit given SHA or branch?
carolinaecalderon Oct 9, 2024
5685d65
I think I'm correctly reading from version file now
carolinaecalderon Oct 11, 2024
2f46c61
replacing w anchors
carolinaecalderon Oct 11, 2024
8d23145
tmp
carolinaecalderon Oct 11, 2024
53ced7a
moving to jobs
carolinaecalderon Oct 11, 2024
81a4861
adding image
carolinaecalderon Oct 11, 2024
b6898d8
adding escape
carolinaecalderon Oct 11, 2024
dfbcaf5
removing yaml_file
carolinaecalderon Oct 11, 2024
7fd3312
fixing i hope
carolinaecalderon Oct 11, 2024
b44fb33
one more
carolinaecalderon Oct 11, 2024
5841479
more edits
carolinaecalderon Oct 11, 2024
90b55c2
wip
carolinaecalderon Oct 14, 2024
e8929d1
remove parameters
carolinaecalderon Oct 14, 2024
61c73ea
wip
carolinaecalderon Oct 14, 2024
2dc4b58
wip3
carolinaecalderon Oct 14, 2024
a304f9d
wip
carolinaecalderon Oct 14, 2024
106c553
adding parameter
carolinaecalderon Oct 14, 2024
75d3ebf
lets pass in the sha
carolinaecalderon Oct 14, 2024
4059e5b
add to requires
carolinaecalderon Oct 14, 2024
b14e9cf
add upstream
carolinaecalderon Oct 14, 2024
37ee4ea
just trying to directly rebuild
carolinaecalderon Oct 14, 2024
4c9f46d
wip
carolinaecalderon Oct 14, 2024
e85d98d
wip
carolinaecalderon Oct 14, 2024
192e853
typo
carolinaecalderon Oct 14, 2024
abc4db5
remove license-gen
carolinaecalderon Oct 14, 2024
2cdf9e7
restructuring
carolinaecalderon Oct 16, 2024
d28a813
remove det version from install wheel
carolinaecalderon Oct 18, 2024
e53a404
rbac tests working! todo; clean up & schedule
carolinaecalderon Oct 18, 2024
26493c1
add schedule?
carolinaecalderon Oct 18, 2024
86a26a4
maybe the filters can be wrong?
carolinaecalderon Oct 18, 2024
50d1baf
apparently filters are essential
carolinaecalderon Oct 18, 2024
8066956
adding references to upgrade_tests scheduled pipeline
carolinaecalderon Oct 22, 2024
1f6d6cd
remove reference to circleCI filter -- let circleCI UI do it from now on
carolinaecalderon Oct 22, 2024
f867d1d
redoing this
carolinaecalderon Oct 22, 2024
955a400
debug
carolinaecalderon Oct 22, 2024
5fe0ca3
typo
carolinaecalderon Oct 22, 2024
41b3332
typo
carolinaecalderon Oct 22, 2024
0a347e0
wip
carolinaecalderon Oct 22, 2024
098b990
test this works
carolinaecalderon Oct 22, 2024
e1d5d87
debug
carolinaecalderon Oct 22, 2024
50c5286
wip
carolinaecalderon Oct 22, 2024
c07f38f
removing trigger in case
carolinaecalderon Oct 22, 2024
a18b476
wip
carolinaecalderon Oct 22, 2024
df5d971
wip
carolinaecalderon Oct 22, 2024
b02df5f
replace with anchors
carolinaecalderon Oct 22, 2024
73b6201
wip
carolinaecalderon Oct 22, 2024
1563c89
wip
carolinaecalderon Oct 22, 2024
6c045c7
add origin tracking
carolinaecalderon Oct 22, 2024
2d5ec78
edits
carolinaecalderon Oct 23, 2024
7e4febb
wip
carolinaecalderon Oct 23, 2024
6731079
wip
carolinaecalderon Oct 23, 2024
2bb26db
rework
carolinaecalderon Oct 23, 2024
4a0145c
changing test versions
carolinaecalderon Oct 23, 2024
7397f90
edit
carolinaecalderon Oct 23, 2024
f53e051
wip
carolinaecalderon Oct 23, 2024
ab97acc
pls work
carolinaecalderon Oct 23, 2024
4066136
readding make-component
carolinaecalderon Oct 23, 2024
daebed6
removing pip install
carolinaecalderon Oct 23, 2024
8a4cd70
testing if this wroks
carolinaecalderon Oct 23, 2024
730e081
getting rid of sleep
carolinaecalderon Oct 23, 2024
582cf15
final wip i hope
carolinaecalderon Oct 23, 2024
4def26c
undoing extra changes
carolinaecalderon Oct 23, 2024
956d8f5
removing managed devcluster e2e from upgrade tests because they need …
carolinaecalderon Oct 23, 2024
3bc4c97
adding license-gen for failing tests?
carolinaecalderon Oct 24, 2024
b5c56b9
testing adding read context for fix
carolinaecalderon Oct 24, 2024
7a20e7e
adding read context to the rest
carolinaecalderon Oct 25, 2024
e6c6f46
changing to 0.37.0
carolinaecalderon Oct 28, 2024
a7e0038
rename & add run test
carolinaecalderon Oct 29, 2024
06fd8db
fix indentation
carolinaecalderon Oct 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,26 @@ jobs:
jq '. += {"do_nightly_tests": true}' < "<<pipeline.parameters.params_basename>>" > tmpfile
mv -v tmpfile "<<pipeline.parameters.params_basename>>"
fi
- run:
name: Check for scheduled upgrade run.
command: |
if [[ "scheduled_pipeline" == "<<pipeline.trigger_source>>"
&& "upgrade_tests" == "<<pipeline.schedule.name>>"
]]
then
# add the parameter(s) to the temp file
jq '. += {"do_upgrade_tests": true}' < "<<pipeline.parameters.params_basename>>" > tmpfile
mv -v tmpfile "<<pipeline.parameters.params_basename>>"
fi
- run:
name: Check for labeled upgrade run.
command: |
if gh pr view --json labels --jq ".labels[].name" \
| grep -w "ci-run-upgrade-tests"
then
jq '. += {"do_upgrade_tests": true}' < "<<pipeline.parameters.params_basename>>" > tmpfile
mv -v tmpfile "<<pipeline.parameters.params_basename>>"
fi

- run:
name: Set det-version parameter.
Expand Down
278 changes: 278 additions & 0 deletions .circleci/real_config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,9 @@ parameters:
do_nightly_tests:
type: boolean
default: false
do_upgrade_tests:
type: boolean
default: false
ee:
type: boolean
default: false
Expand Down Expand Up @@ -98,6 +101,10 @@ release-dryrun: &release-dryrun
# Tags like: v3.1.3+dryrun
- /v\d+\.\d+\.\d+\+dryrun/

release-upgrade: &release-upgrade
carolinaecalderon marked this conversation as resolved.
Show resolved Hide resolved
# This value should be updated on every release to capture the updated supported upgrades.
- "release-0.37.0"

upstream-feature-branch: &upstream-feature-branch
branches:
ignore:
Expand Down Expand Up @@ -1487,6 +1494,11 @@ commands:
command: devcluster --oneshot -c .circleci/devcluster/<<parameters.devcluster-config>> --target-stage <<parameters.target-stage>> || circleci-agent step halt
background: true

stop-devcluster:
steps:
- run:
command: pkill -SIGHUP -f "devcluster" && rm -rf /tmp/devcluster/lock

make-component:
description: "Basic parameterized version of make for building Determined components."
parameters:
Expand Down Expand Up @@ -3170,6 +3182,9 @@ jobs:
k8s-version:
type: string
default: "1.29.5"
upgrade-branch:
carolinaecalderon marked this conversation as resolved.
Show resolved Hide resolved
type: string
default: ""
machine:
image: <<pipeline.parameters.machine-image>>
resource_class: <<parameters.resource-class>>
Expand All @@ -3190,6 +3205,17 @@ jobs:
determined: True
extra-requirements-file: "e2e_tests/tests/requirements.txt"
executor: <<pipeline.parameters.machine-image>>

- when:
condition: <<parameters.upgrade-branch>>
steps:
- run:
name: Get Determined version branch commits.
command: |
git fetch origin <<parameters.upgrade-branch>> && git checkout --track origin/<<parameters.upgrade-branch>>
- license-gen
- run: make -C master build
- run: make -C agent build
carolinaecalderon marked this conversation as resolved.
Show resolved Hide resolved

- when:
condition: <<parameters.run-minikubes>>
Expand All @@ -3210,6 +3236,21 @@ jobs:
- run:
name: Start additionalrm minikube
command: minikube start --profile additionalrm --kubernetes-version <<parameters.k8s-version>>
- when:
condition: <<parameters.upgrade-branch>>
steps:
- install-devcluster
- unless:
condition: <<parameters.managed-devcluster>>
steps:
- start-devcluster:
devcluster-config: <<parameters.devcluster-config>>
target-stage: <<parameters.target-stage>>
- stop-devcluster
- checkout
- run: make -C master build
- run: make -C agent build

- install-devcluster
- unless:
condition: <<parameters.managed-devcluster>>
Expand Down Expand Up @@ -4721,6 +4762,243 @@ workflows:
requires:
- package-and-push-system-dev-ee

test-e2e-upgrade:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm not sure i understand the value in these tests. if i understand, each e2e run runs determined $OLD_VERSION, doesn't do anything on that version, stops it, upgrades to the $HEAD, and then runs a test suite. but i don't see much of a difference between that and just testing against $HEAD to begin with; the only difference between those systems would be that one took a brief pause at some DB migration number, but even then they all the same migrations in the same order with essentially the same data. if we created some data or an experiment on $OLD_VERSION then upgraded to $HEAD and made sure the data looked right or the experiment successfully unpaused, that would cover a lot more of our previously experienced upgrade bugs.

when: << pipeline.parameters.do_upgrade_tests >>
jobs:
- build-proto
- build-helm
- build-react:
dev-mode: true
- build-docs:
requires:
- build-helm
- build-proto
- build-go:
name: build-go-ee
context: github-read

- test-e2e:
name: test-e2e-cpu
context:
- dev-ci-cluster-default-user-credentials
- github-read
requires:
- build-go-ee
parallelism: 6
resource-class: xlarge
tf2: true
mark: e2e_cpu
matrix:
parameters:
upgrade-branch: *release-upgrade

- test-e2e:
name: test-e2e-cpu-double
context:
- dev-ci-cluster-default-user-credentials
- github-read
requires:
- build-go-ee
parallelism: 5
resource-class: large
tf2: true
mark: e2e_cpu_2a
target-stage: agent2
devcluster-config: double-priority.devcluster.yaml
matrix:
parameters:
upgrade-branch: *release-upgrade

- test-e2e:
name: test-e2e-cpu-oauth
context:
- dev-ci-cluster-default-user-credentials
- github-read
requires:
- build-go-ee
parallelism: 1
devcluster-config: oauth.devcluster.yaml
mark: test_oauth
target-stage: agent1
matrix:
parameters:
upgrade-branch: *release-upgrade

- test-e2e:
name: test-e2e-multi-k8s
context:
- dev-ci-cluster-default-user-credentials
- github-read
requires:
- build-go-ee
parallelism: 1
tf2: true
mark: e2e_multi_k8s
target-stage: master
devcluster-config: multi-k8s.devcluster.yaml
run-minikubes: true
multi-k8s: true
matrix:
parameters:
upgrade-branch: *release-upgrade

- test-e2e:
name: test-e2e-single-k8s
context:
- dev-ci-cluster-default-user-credentials
- github-read
requires:
- build-go-ee
parallelism: 1
mark: e2e_single_k8s
target-stage: master
devcluster-config: single-k8s.devcluster.yaml
run-minikubes: true
matrix:
parameters:
upgrade-branch: *release-upgrade

- test-e2e:
name: test-e2e-port-registry
context:
- dev-ci-cluster-default-user-credentials
- github-read
requires:
- build-go-ee
parallelism: 1
devcluster-config: port-registry.devcluster.yaml
mark: port_registry
target-stage: agent
matrix:
parameters:
upgrade-branch: *release-upgrade

- test-e2e:
name: test-e2e-cpu-elastic
context:
- dev-ci-cluster-default-user-credentials
- github-read
requires:
- build-go-ee
parallelism: 1
resource-class: large
mark: e2e_cpu_elastic
devcluster-config: elastic.devcluster.yaml
target-stage: agent
matrix:
parameters:
upgrade-branch: *release-upgrade

- test-e2e:
name: test-e2e-postgres10-with-ssl
context:
- dev-ci-cluster-default-user-credentials
- github-read
requires:
- build-go-ee
parallelism: 1
mark: e2e_cpu_postgres
devcluster-config: postgres-with-ssl.devcluster.yaml
target-stage: agent
postgres-version: "10"
matrix:
parameters:
upgrade-branch: *release-upgrade

- test-e2e:
name: test-e2e-postgres14-with-ssl
context:
- dev-ci-cluster-default-user-credentials
- github-read
requires:
- build-go-ee
parallelism: 1
mark: e2e_cpu_postgres
devcluster-config: postgres-with-ssl.devcluster.yaml
target-stage: agent
postgres-version: "14"
matrix:
parameters:
upgrade-branch: *release-upgrade

- test-e2e:
name: test-e2e-old-agent-versions
context:
- dev-ci-cluster-default-user-credentials
- github-read
requires:
- build-go-ee
parallelism: 1
mark: e2e_cpu_postgres
devcluster-config: custom-agent-version.devcluster.yaml
target-stage: agent
matrix:
parameters:
upgrade-branch: *release-upgrade
agent-version: ["0.17.10"]

- test-e2e:
name: test-e2e-agent-connection-loss
context:
- dev-ci-cluster-default-user-credentials
- github-read
requires:
- build-go-ee
parallelism: 1
mark: e2e_cpu_agent_connection_loss
devcluster-config: agent-no-connection.devcluster.yaml
target-stage: agent
wait-for-master: false
extra-pytest-flags: "--no-compare-stats"
collect-det-job-logs: false
matrix:
parameters:
upgrade-branch: *release-upgrade

- test-e2e:
name: test-e2e-saml
context:
- okta
- dev-ci-cluster-default-user-credentials
- github-read
requires:
- build-go-ee
parallelism: 1
devcluster-config: saml.devcluster.yaml
mark: e2e_saml
matrix:
parameters:
upgrade-branch: *release-upgrade

- test-e2e:
name: test-e2e-rbac
context:
- dev-ci-cluster-default-user-credentials
- github-read
requires:
- build-go-ee
parallelism: 1
mark: e2e_cpu_rbac
devcluster-config: single-rbac.devcluster.yaml
matrix:
parameters:
upgrade-branch: *release-upgrade

- test-e2e:
name: test-e2e-cpu-model-registry-rbac
context:
- dev-ci-cluster-default-user-credentials
- github-read
requires:
- build-go-ee
parallelism: 1
devcluster-config: rbac-model-registry.yaml
mark: test_model_registry_rbac
target-stage: agent1
matrix:
parameters:
upgrade-branch: *release-upgrade

test-e2e-longrunning:
<<: *do-not-run-on-manual-trigger
jobs:
Expand Down
Loading