Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Database and ObjectStorage Health Checks #133

Conversation

gmfrasca
Copy link
Member

@gmfrasca gmfrasca commented Jun 5, 2023

Fixes #149

  • Delays deploying DSPA Manifests until DB and ObjStore Connections are healthy

Description

  • Performs simple health checks on the database and object storage connections
  • Prevents DSPO from deploying DSPA components (apiserver, persistenceagent, etc) until conns are healthy
  • Introduces DatabaseAvailable and ObjectStoreAvailable readiness metrics

How Has This Been Tested?

Notes to reviewer:

  • In it's current state, this makes 'local' runs of the DSPO difficult/impossible, as the Object Store host references internal Services which are not available on an external-running DSPO
  • Do we want to add an override mechanism here?
  • Update: a spec.database.disableHealthCheck field and funtionality was added to the DSPA CRD to enable local runs

Merge criteria:

  • The commits are squashed in a cohesive manner and have meaningful messages.
  • Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
  • The developer has manually tested the changes and verified that the changes work

@gmfrasca gmfrasca force-pushed the storage-readiness-conditions branch from 3ec6a98 to 6992e9b Compare July 7, 2023 20:16
@gmfrasca gmfrasca force-pushed the storage-readiness-conditions branch 2 times, most recently from 10478fe to b4fba5b Compare July 10, 2023 21:58
@gmfrasca gmfrasca changed the title WIP: Add Database and ObjectStorage Health Checks Add Database and ObjectStorage Health Checks Jul 10, 2023
controllers/database.go Outdated Show resolved Hide resolved
controllers/database.go Outdated Show resolved Hide resolved
controllers/storage.go Outdated Show resolved Hide resolved
controllers/storage.go Outdated Show resolved Hide resolved
@gmfrasca gmfrasca force-pushed the storage-readiness-conditions branch 2 times, most recently from ec253e2 to dc4c2ab Compare July 19, 2023 21:39
- Delays deploying DSPA Manifests until DB and ObjStore
  Connections are healthy
- Introduces status condition of "ComponentDeploymentNotFound" for
  Required Components with missing Deployments
- Useful for local testing and debugging
@gmfrasca gmfrasca force-pushed the storage-readiness-conditions branch from dc4c2ab to 0563af3 Compare July 19, 2023 22:08
@HumairAK HumairAK added the qe/verify Labels to inform qe on issues to verify. label Jul 24, 2023
- specifying Spec.Database.DsiableHealthCheck but not
  externalDB/mariaDB leads to an issue where the DSPO does not
  recognize the need for a default DB deployment.  This handles
  that case by including it in the default-deploy behavior
controllers/database.go Outdated Show resolved Hide resolved
controllers/dspipeline_controller.go Outdated Show resolved Hide resolved
controllers/database.go Show resolved Hide resolved
controllers/database.go Outdated Show resolved Hide resolved
controllers/dspipeline_controller.go Outdated Show resolved Hide resolved
api/v1alpha1/dspipeline_types.go Show resolved Hide resolved
@HumairAK
Copy link
Contributor

HumairAK commented Jul 25, 2023

functionality wise this lgtm, just a few minor changes above

@HumairAK
Copy link
Contributor

/lgtm

@openshift-ci openshift-ci bot added the lgtm label Jul 25, 2023
Copy link
Contributor

@amadhusu amadhusu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@rimolive
Copy link
Contributor

/approve

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 26, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amadhusu, rimolive

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved lgtm qe/verify Labels to inform qe on issues to verify.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Logic to DSP Operator to wait for s3/db to be ready before deploying
5 participants