Rollback

How deployment works

In line with the GitOps methodology, the running system is a reflection of a "single source of truth": the config we have on GitHub. With the system dynamically changing to reflect whatever the config says, deployments and rollbacks are done by changing that config with a Pull Request.

The deployment

After a merge, tests are run and assuming they pass, an image is built and pushed to our image registry. Flux runs inside our cluster and watches our image registry for new images. When it sees a new image, it changes the single source of truth by making a commit back to master that looks like this.

The content of the commit is a change to the image tag in our Kustomize config.

Regenerating our config (as Flux does inside the cluster), now produces config saying that Kubernetes should be running gcr.io/track-compliance/api-js:master-26a4b87 instead of gcr.io/track-compliance/api-js:master-f01225d, triggering a rolling deployment of that image in Kubernetes.

Reverting a deployment

For those moments when things don't work as planned, we can roll back our changes using the same process we used to introduce the change in the first place: a Pull Request.

Assuming the deployment of that new image introduced a bug, we can revert to the previous image by reverting this commit.

Find the commit

Find the commit that deployed the image.

[mike@ouroboros tracker]$ git show d4cc490
commit d4cc49065c73b72b4879877e170fbc14120af97c (HEAD -> master, origin/master, origin/HEAD)
Author: fluxcd <fluxcd@users.noreply.github.com>
Date:   Mon Jan 11 15:07:07 2021 +0000

    Auto-release gcr.io/track-compliance/api-js:master-26a4b87
    
    [ci skip]

diff --git a/app/bases/kustomization.yaml b/app/bases/kustomization.yaml
index ca43c269..f48e625e 100644
--- a/app/bases/kustomization.yaml
+++ b/app/bases/kustomization.yaml
@@ -4,7 +4,7 @@ images:
 - name: gcr.io/track-compliance/api
   newTag: master-4003aac
 - name: gcr.io/track-compliance/api-js
-  newTag: master-f01225d
+  newTag: master-26a4b87
 - name: gcr.io/track-compliance/autoscan
   newTag: master-0000000
 - name: gcr.io/track-compliance/db-migration

Make a branch

Make a branch (the name doesn't really matter).

[mike@ouroboros tracker]$ git checkout -b rollback-d4cc490
Switched to a new branch 'rollback-d4cc490'

Revert

Use git revert to invert the commit.

[mike@ouroboros tracker]$ git revert d4cc490
[rollback-d4cc490 980f6cbe] Revert "Auto-release gcr.io/track-compliance/api-js:master-26a4b87"
 1 file changed, 1 insertion(+), 1 deletion(-)
[mike@ouroboros tracker]$ git show
commit 980f6cbe317cff69d40fb76c72d8fa9e4aceb99d (HEAD -> rollback-d4cc490)
Author: Mike Williamson <mike@korora.ca>
Date:   Mon Jan 11 11:11:57 2021 -0500

    Revert "Auto-release gcr.io/track-compliance/api-js:master-26a4b87"
    
    This reverts commit d4cc49065c73b72b4879877e170fbc14120af97c.

diff --git a/app/bases/kustomization.yaml b/app/bases/kustomization.yaml
index f48e625e..ca43c269 100644
--- a/app/bases/kustomization.yaml
+++ b/app/bases/kustomization.yaml
@@ -4,7 +4,7 @@ images:
 - name: gcr.io/track-compliance/api
   newTag: master-4003aac
 - name: gcr.io/track-compliance/api-js
-  newTag: master-26a4b87
+  newTag: master-f01225d
 - name: gcr.io/track-compliance/autoscan
   newTag: master-0000000
 - name: gcr.io/track-compliance/db-migration

Push your branch

This puts up a new branch that can be used for a PR, following the normal review process.

[mike@ouroboros tracker]$ git push origin rollback-d4cc490

What's next

This process fixes the system by telling Kubernetes to run an older image. The bug still needs to be tracked down and fixed before any more deployments happen. Running docker run gcr.io/track-compliance/api-js:master-26a4b87 (plus whatever env and port options) can help reproduce the error locally.