Service Management in Bedrock

This document describes the workflow for deploying a set of services Bedrock workflows via the spk CLI tool.

High Level Overview

Confirm you have met the Requirements for SPK's automation.
Install and configure SPK
Create and configure the required high level definition and manifest repositories
Initialize the Bedrock Application Repository
Add a service to the Bedrock Application Repository
Optional: Create a Service Revision

Notes:

Steps 1-4 typically only need to be done once. Multiple clusters may be configured to sync from the single Materialized Manifest Repositories, and multiple Project repositories can be pointed to the single High Level Definition Repository.
Step 5 can be repeated each time you need to onboard a service to your Bedrock automated infrastructure.
Step 6 can be run as many times as required to add a service revision to a Bedrock project.

An overview of how these different pieces fit together from an automation perspective:

Requirements

This guide assumes a few things as requirements to use this automation:

The application code and supporting repositories are hosted on Azure Devops.
- If starting from scratch, then first create a new Azure Devops Organization, then create a project.
The application will be packaged and run using container images hosted on Azure Container Registry
The user running spk has full access to the above resources.
The user is running the latest spk release.
The user has Azure CLI installed.
The user is running git version 2.22 or later.
Optional: If a user wishes to store helm charts in the application repositories, then all repositories (application, high level definition, materialized manifests) must be in the same Azure DevOps Organization AND Project.

Installing and Configuring SPK

spk is the Command Line Interface that provides automation around defining and operating Kubernetes clusters with Bedrock principles.

Setup SPK

Download the latest version of spk from the releases page and add it to your PATH.

To setup a local configuration:

Generate a Personal Access Token
Create a spk config file
Initialize spk

Generate Personal Access Token

Generate a new Personal Access Token (PAT) to grant spk permissions in the Azure Devops Project. Please grant PAT the following permissions:

Build (Read & execute)
Code (Read, write, & manage)
Variable Groups (Read, create, & manage)

For help, follow the guide.

Create SPK config file

Create a copy of spk-config.yaml from the starter template with the appropriate values for the azure_devops section.

Note: This spk-config.yaml should not be commited anywhere, as it contains sensitive credentials.

Initialize SPK

Run spk init -f <spk-config.yaml> where <spk-config.yaml> the path to the configuation file.

Note: When running spk init -f <spk-config.yaml>, spk will copy the values from the config file and store it into local memory elsewhere. If you wish to utilize spk with another project or target, then you must rerun spk init with another configuration first OR, you may overwrite each commands via flags.

Repositories

Our next step is to create or onboard the repositories that support the deployment of our services:

The high level definition repository
The materialized manifest repository
The application source code repository

High Level Definition Repository

This repository holds the Bedrock High Level Deployment Definition (HLD) and associated configurations.

This HLD is processed via fabrikate in Azure Devops on each change to generate Kubernetes YAML manifests that are applied to the Kubernetes cluster by Flux.

Initializing the High Level Definition Repository

Create a repository in the given AzDO project.
Edit your SPK config to point to this repo (if you haven't already done this).
Clone the repository.
Initialize via spk, this will add the fabrikate traefik2 as the initial sample component. This can be overridden via optional flags.
```
spk hld init --git-push
```

NOTE spk hld command documentation can be found here.

Materialized Manifests Repository

This repository holds all the materialized kubernetes manifests that should be deployed to a cluster. If a cluster has been deployed via Bedrock's Terraform templates, then flux should be configured to point to this repository and will deploy all manifests in this repository to the cluster periodically.

Initializing the Materialized Manifests Repository

Create a repository in the given AzDO project.
Edit your SPK config to point to this repo (if you haven't already done this).
Clone the repository.

Add a simple README to the repository

touch README.md
echo "This is the Flux Manifest Repository." >> README.md
git add -A
git commit -m "Initializing Materialized Manifests repository with a README."
git push -u origin --all

Deploy Manifest Generation Pipeline

Deploy a manifest generation pipeline between the high level definition repo and the materialized manifests repo. Assuming you have configured spk, you can run this without flag parameters from your HLD repo root:

$ spk hld install-manifest-pipeline

Application Repositories

These repositories hold the application code, its associated Dockerfile(s), and helm deployment charts.

Additionally, these repositories can hold one (single application) or more (monorepository) applications depending on your development methodology. Typically, each repository shold be configured with a "hld-lifecycle" Azure DevOps pipeline that will add all managed applications inside the repository to the High Level Definition Repository. Additionally, each application inside the repository should also have an associated Azure DevOps multi-stage pipeline that both builds and deploys the latest Docker image to Azure Container Registry and updates the associated configuation in the HLD repository with the latest image tag.

-TBD Section on packages directory and manging monorepositories vs single application repositories

Initializing an Application Repository

Create a repository in the given AzDO project.
Clone the repository.

Initialize the project via spk

$ spk project init
$ git add -A
$ git commit -m "Initializing application repository."
$ git push -u origin --all

Create a variable group via spk. If you are using a repo per service source control strategy, you only need to do this once.
```
$ export VARIABLE_GROUP_NAME=<my-vg-name>
$ spk project create-variable-group $VARIABLE_GROUP_NAME -r $ACR_NAME -u $SP_APP_ID -t $SP_TENANT -p $SP_PASS
$ git add -A
$ git commit -m "Adding Project Variable Group."
$ git push -u origin --all
```
where ACR_NAME is the name of the Azure Container Registry where the Docker Images will be served from and SP_APP_ID, SP_PASS, and, SP_TENANT are an associated Service Principal's ID, Password, and Tenant, that have Read and Write access to the ACR.
Deploy the lifecycle pipeline (optional flag parameters can be used if spk was not intialized)
```
$ spk project install-lifecycle-pipeline --org-name $ORG_NAME --devops-project $DEVOPS_PROJECT --repo-url $SERVICE_REPO_URL --repo-name $SERVICE_NAME
```
where ORG_NAME is the name of your Azure Devops org, DEVOPS_PROJECT is the name of your Azure Devops project, SERVICE_REPO_URL is the url that you used to clone your service from Azure Devops, and SERVICE_NAME is the name of the service.

Note: If you are using a repo per service source control strategy you should run install-lifecycle-pipeline once for each repo.

NOTE spk project command documentation can be found here.

Adding a Service to a Application Repository

Clone the repository.
Create the service via spk, there are optional parameters that should be used to configure the service and its associated helm charts (other optional flag parameters can be used if spk was not intialized)
```
SERVICE_NAME=<my-new-service-name>
spk service create $SERVICE_NAME . ...
git add -A
git commit -m "Adding $SERVICE_NAME to the repository."
git push -u origin --all
```

Deploy the service's multistage build pipeline via spk (optional flag parameters can be used if spk was not intialized)

spk service install-build-pipeline . -n $SERVICE_NAME-build-pipeline -o $ORG_NAME -r $SERVICE_NAME -u $SERVICE_REPO_URL -d $DEVOPS_PROJECT

Review and accept the pull request to add the service to the high level definition in Azure Devops.

NOTE spk service command documentation can be found here.

Helm Configuration for SPK

spk service create allows a user to configure a service a number of ways with a backing helm chart.

Presently, there are are a number of options for spk service create documented below:

  -c, --helm-chart-chart <helm-chart>                         bedrock helm chart name. --helm-chart-* and --helm-config-* are exclusive; you may only use one. (default: "")
  -r, --helm-chart-repository <helm-repository>               bedrock helm chart repository. --helm-chart-* and --helm-config-* are exclusive; you may only use one. (default: "")

  -g, --helm-config-git <helm-git>                            bedrock helm chart configuration git repository. --helm-chart-* and --helm-config-* are exclusive; you may only use one. (default: "")
  -b, --helm-config-branch <helm-branch>                      bedrock custom helm chart configuration branch. --helm-chart-* and --helm-config-* are exclusive; you may only use one. (default: "")
  -p, --helm-config-path <helm-path>                          bedrock custom helm chart configuration path. --helm-chart-* and --helm-config-* are exclusive; you may only use one. (default: "")

As noted by the the documentation text, helm-chart-* and helm-config-* are both mutually exclusive configurations: you can only use one set of configurations or the other.

This section intends on documenting the various use cases for both sets of mutually exclusive configurations.

Helm Charts in a well-known Helm Repository

A Helm Repository is a well known set of helm charts conforming to the helm repository guidelines. Perhaps the best known helm repository is the community run helm charts repository.

As an spk user, if you would like to incorporate helm charts from a well known public repository, you may simply run spk the following helm-chart arguments:

spk service create nginx my-nginx-service --helm-chart-chart stable/nginx --helm-chart-repository github.com/helm/charts

Helm Charts in a distinct Git Repository from Application Sources in the same Azure DevOps Project

If your Helm Charts are in their own distinct Git Repository in the same Azure DevOps project, you can use the helm-config arguments to configure spk:

spk service create fabrikam path/to/fabrikam \
  --helm-config-git https://dev.azure.com/fabrikam/fabrikam-project/_git/fabrikam-helm-charts
  --helm-config-branch master \
  --helm-path /charts/fabrikam

The above invocation presumes that the helm chart repository configured for spk is different from the application repository configured for spk usage.

The helm-config-git parameter must not contain the username portion of a url. If you retrieve the URL from Azure DevOps's "Clone Repository" UI, it will automatically have the username filled for an HTTPs clone ie:

https://fabrikam@dev.azure.com/fabrikam/fabrikam-project/_git/fabrikam-helm-charts

Ensure that the you remove the fabrikam@ portion of the URL when passing parameters to spk service create --helm-config-git:

https://dev.azure.com/fabrikam/fabrikam-project/_git/fabrikam-helm-charts

When you invoke spk service create with --helm-config argments, there is a 4th optional argument that can be provided --helm-config-access-token-variable. This configuration option is the name of the environment variable containing a Personal Access Token to access the git repository in helm config git. In this scenario, however, both git repositories are in the same Azure DevOps project, fabrikam-project, so it is un-necessary to provide --helm-config-access-token-variable, as the Personal Access Token utilized by the pipelines, ACCESS_TOKEN_SECRET, will be the same for both application repository and helm chart repository. When --helm-config-access-token-variable is not provided, the repository defined by --helm-config-git will automatically use ACCESS_TOKEN_SECRET. The following scenario covers a situation in which you will want to configure the --helm-config-access-token-variable to something custom.

Helm Charts in a distinct Git Repository from Application Sources in a different Azure DevOps Project

If your Helm Charts are in their own distinct Git Repository in a different Azure DevOps project, you can use still use the helm-config arguments to configure spk, but must also provide another option, --helm-config-access-token-variable. This configuration option is the name of the environment variable containing the Personal Access Token to access the git repository in helm-config-git:

spk service create fabrikam path/to/fabrikam \
  --helm-config-git https://dev.azure.com/fabrikam/fabrikam-helm-charts-project/_git/fabrikam-helm-charts \
  --helm-config-branch master \
  --helm-path /charts/fabrikam \
  --helm-config-access-token-variable FABRIKAM_HELM_CHARTS_REPO_PAT

The helm-config-git parameter must not contain the username portion of a url. If you retrieve the URL from Azure DevOps's "Clone Repository" UI, it will automatically have the username filled for an HTTPs clone ie:

https://fabrikam@dev.azure.com/fabrikam/fabrikam-helm-charts-project/_git/fabrikam-helm-charts

Ensure that the you remove the fabrikam@ portion of the URL when passing parameters to spk service create --helm-config-git:

https://dev.azure.com/fabrikam/fabrikam-helm-charts-project/_git/fabrikam-helm-charts

Note the different Azure DevOps URL for the helm charts project in Azure DevOps and the addition of the --helm-config-access-token-variable parameter.

The --helm-config-access-token-variable parameter configures how an access.yaml file is written to the HLD for fabrikate to consume when rendering helm charts. For more information on authenticating with private git repositories when rendering helm charts, please refer to fabrikate's documentation.

When the bedrock.yaml is committed, all the pipelines created, and the HLD repository is populated, you must add the environment variable, FABRIKAM_HELM_CHARTS_REPO_PAT to the HLD to Materialized pipeline as a pipeline variable. To do so, find the HLD to Materialized pipeline in the pipelines view on Azure DevOps, select it, select Variables, then select New Variable:

Helm Charts in the same repository as the application

If your Helm Charts are intended to be placed adjacent to your application source (no distinct git repository), you may still use the helm-config arguments to configure spk.

If you presume that the fabrikam-app repository is the same repository as the repository of your application sources, then the following invocation will allow a user to configure a service with helm charts tracked in the same repository.

spk service create
  --helm-config-git https://dev.azure.com/fabrikam/fabrikam-project/_git/fabrikam-app \
  --helm-config-branch master \
  --helm-path /charts/fabrikam

The helm-config-git parameter must not contain the username portion of a url. If you retrieve the URL from Azure DevOps's "Clone Repository" UI, it will automatically have the username filled for an HTTPs clone ie:

https://fabrikam@dev.azure.com/fabrikam/fabrikam-project/_git/fabrikam-helm-charts

Ensure that the you remove the fabrikam@ portion of the URL when passing parameters to spk service create --helm-config-git:

https://dev.azure.com/fabrikam/fabrikam-project/_git/fabrikam-helm-charts

Creating a Service Revision

Create and checkout a new git branch

git branch <my-new-feature-branch>
git checkout <my-new-feature-branch>

Make code changes and commit

echo "# My New Added File" >> myNewFile.md
git add myNewFile.md
git commit -m "Adding my new file"
git push --set-upstream origin <my-new-feature-branch>

Create Service Revision via spk (optional flag parameters can be used if spk was not intialized)

spk service create-revision . -n $SERVICE_NAME-build-pipeline -o $ORG_NAME -r $SERVICE_NAME -u $SERVICE_REPO_URL -d $DEVOPS_PROJECT

NOTE spk service command documentation can be found here.

Helm Charts

To have spk's build pipelines work properly, an application needs an associated Helm chart with specific variables.

See the guide to building helm charts with spk for more details

Requirements

TBD

Sample Helm Chart

This repository bundles a sample helm chart. Please refer to the guide for building helm charts and the sample helm chart for details.

Variable Groups

TBD

Done to hold secure credentials and secrets.

Pipelines

TBD

Application build & update (1 per application)
HLD lifecycle (adds applications to HLD repo)
HLD to Manifests (generates manifests via fabrikate and places manifests into flux's source repo)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

project-service-management-guide.md

project-service-management-guide.md

Service Management in Bedrock

High Level Overview

Requirements

Installing and Configuring SPK

Setup SPK

Generate Personal Access Token

Create SPK config file

Initialize SPK

Repositories

High Level Definition Repository

Initializing the High Level Definition Repository

Materialized Manifests Repository

Initializing the Materialized Manifests Repository

Deploy Manifest Generation Pipeline

Application Repositories

Initializing an Application Repository

Adding a Service to a Application Repository

Helm Configuration for SPK

Helm Charts in a well-known Helm Repository

Helm Charts in a distinct Git Repository from Application Sources in the same Azure DevOps Project

Helm Charts in a distinct Git Repository from Application Sources in a different Azure DevOps Project

Helm Charts in the same repository as the application

Creating a Service Revision

Helm Charts

Requirements

Sample Helm Chart

Variable Groups

Pipelines

Files

project-service-management-guide.md

Latest commit

History

project-service-management-guide.md

File metadata and controls

Service Management in Bedrock

High Level Overview

Requirements

Installing and Configuring SPK

Setup SPK

Generate Personal Access Token

Create SPK config file

Initialize SPK

Repositories

High Level Definition Repository

Initializing the High Level Definition Repository

Materialized Manifests Repository

Initializing the Materialized Manifests Repository

Deploy Manifest Generation Pipeline

Application Repositories

Initializing an Application Repository

Adding a Service to a Application Repository

Helm Configuration for SPK

Helm Charts in a well-known Helm Repository

Helm Charts in a distinct Git Repository from Application Sources in the same Azure DevOps Project

Helm Charts in a distinct Git Repository from Application Sources in a different Azure DevOps Project

Helm Charts in the same repository as the application

Creating a Service Revision

Helm Charts

Requirements

Sample Helm Chart

Variable Groups

Pipelines