page_type	languages	products	description	urlFragment
sample	java	Azure Kubernetes Service	End-to-end experience for Java apps in Azure Kubernetes Service (AKS)	java-on-aks

End-to-end experience - Java on AKS

This guide walks you through how to deploy and manage Java apps on the Azure Kubernetes Service.

What will you experience

You will:

• Build Piggymetrics - build a proof-of-concept application, which demonstrates micro service architecture pattern using Spring Boot and Spring Cloud
• Create Mongodb and RabbitMQ on Azure
• Create Azure Kubernetes Service and Azure Container Registry
• Deploy Piggymetrics to Azure Kubernetes Service
• Troubleshoot Java apps in Azure Kubernetes Service
• Automate and rapidly deploy changes to Azure Kubernetes Service - using GitHub Actions or Azure Pipelines
• Rapidly deploy changes to Azure Kubernetes without disruption - blue-green deployments
• Scale out Java apps in Azure Kubernetes Service
• Learn about next steps to production

What you will need

In order to deploy a Java Web app to cloud, you need an Azure subscription. If you do not already have an Azure subscription, you can activate your MSDN subscriber benefits or sign up for a free Azure account.

In addition, you will need the following:

| Azure CLI | Java 8 | Maven 3 | Git | ACR Docker Credential Helper | Docker | Kubectl | Helm | dos2unix |

IMPORTANT - Start Here

Clone this GitHub repo and change directory:

git clone https://github.com/Azure-Samples/java-on-aks.git

cd java-on-aks

Create MongoDB and RabbitMQ

You can create MongoDB and RabbitMQ on Azure by following steps outlined here and capture MongoDB and RabbitMQ coordinates and credentials in setup-env-variables-azure.sh.

Create Azure Container Registry and Azure Kubernetes Service

Create Azure Container Registry

If you have not yet setup the development environment, make a copy of the setup environment variables bash script:

cp .scripts/setup-env-variables-azure-template.sh .scripts/setup-env-variables-azure.sh

Prep the dev environment by populating environment variables in .scripts/setup-env-variables-azure.sh bash script:

# ====== Piggy Metrics Azure Coordinates
export RESOURCE_GROUP=INSERT-your-resource-group-name
export REGION=westus2
export AKS_CLUSTER=INSERT-your-AKS-cluster-name
export CONTAINER_REGISTRY=INSERT-your-Azure-Container-Registry-name

## ===== Mongo DB
export MONGODB_DATABASE=INSERT-your-mongodb-database-name
export MONGODB_USER=INSERT-your-cosmosdb-account-name
export MONGODB_URI="INSERT-your-mongodb-connection-string"
export MONGODB_RESOURCE_ID=INSERT-your-mongodb-resource-id

## ===== Rabbit MQ
export RABBITMQ_RESOURCE_GROUP=INSERT-your-rabbitmq-resource-group-name
export RABBITMQ_VM_NAME=INSERT-your-rabbitmq-virtual-machine-name
export RABBITMQ_ADMIN_USERNAME=INSERT-your-rabbitmq-admin-user-name

# Rabbit MQ
export RABBITMQ_HOST=INSERT-your-rabbitmq-host-public-ip-address
export RABBITMQ_PORT=5672
export RABBITMQ_USERNAME=INSERT-your-rabbitmq-username
export RABBITMQ_PASSWORD=INSERT-your-rabbitmq-password

Then, export these environment variables from the java-on-aks directory:

source .scripts/setup-env-variables-azure.sh

Create an Azure Container Registry instance using Azure CLI:

# Create a Resource Group, if you have not created one
az group create --name ${RESOURCE_GROUP} \
    --location ${REGION}
    
# Create Azure Container Registry
az acr create --name ${CONTAINER_REGISTRY} \
    --resource-group ${RESOURCE_GROUP} \
    --sku basic --location ${REGION}
    
# Log into Azure Container Registry
az acr login -n ${CONTAINER_REGISTRY}

Create Azure Kubernetes Service

Create an Azure Kubernetes Service instance and attach the Azure Container Registry using Azure CLI:

az aks create --name ${AKS_CLUSTER} \
    --resource-group ${RESOURCE_GROUP} \
    --location ${REGION} \
    --attach-acr ${CONTAINER_REGISTRY} \
    --node-vm-size Standard_DS3_v2 \
    --node-count 5

Get access credentials for the AKS cluster:

az aks get-credentials --name ${AKS_CLUSTER} \
    --resource-group ${RESOURCE_GROUP}

Auto instrument for monitoring Java apps on Azure Kubernetes Service

If you do not have an instance of Application Insights, see how to create Application Insights.

Download Java agent (download link) for auto instrumenting for monitoring Java apps on an Azure Kubernetes Service.

From a Linux or MacOS terminal, execute init.sh from the release.

source init.sh

Open the generated values.yaml file in an editor and fill up Kubernetes Cluster target namespace, say default, and Application Insights Instrumentation Key. Like this:

namespaces: 
  - target : "default" # kubernetes namespace where the Java apps will be auto-instrumented
    iKey: "c197cf6b-WWWW-XXXX-YYYY-ZZZZZZZZZZZZ" # Instrumentation Key of the receiving Application Insights resource

Install the Java agent for auto instrumentation:

helm install ./helm-<version>.tgz -f values.yaml --generate-name

Deploy Piggymetrics to Azure Kubernetes Service

Build and push container images for micro service apps

Build Java apps, container images and push images to Azure Container Registry using Maven and Jib:

cd config
mvn compile jib:build \
    -Djib.container.environment=CONFIG_SERVICE_PASSWORD=${CONFIG_SERVICE_PASSWORD}

cd ../registry
mvn compile jib:build

cd ../gateway
mvn compile jib:build

cd ../auth-service
mvn compile jib:build

cd ../account-service
mvn compile jib:build \
    -Djib.container.environment=ACCOUNT_SERVICE_PASSWORD=${ACCOUNT_SERVICE_PASSWORD}

cd ../statistics-service
mvn compile jib:build \
    -Djib.container.environment=STATISTICS_SERVICE_PASSWORD=${STATISTICS_SERVICE_PASSWORD}

cd ../notification-service
mvn compile jib:build \
    -Djib.container.environment=NOTIFICATION_SERVICE_PASSWORD=${NOTIFICATION_SERVICE_PASSWORD}

Jib is configured to start from a production-ready distribution of Java. For example:

<plugin>
    <groupId>com.google.cloud.tools</groupId>
    <artifactId>jib-maven-plugin</artifactId>
    <version>1.8.0</version>
    <configuration>
        <from>
            <!-- production-ready distribution of Java -->
            <image>mcr.microsoft.com/java/jre-headless:8u232-zulu-alpine</image>
        </from>
        <to>
            <image>${CONTAINER_REGISTRY}.azurecr.io/${parent.artifactId}-${project.name}</image>
        </to>
        <container>
            <jvmFlags>
                <jvmFlag>-Xms2048m</jvmFlag>
                <jvmFlag>-Xmx2048m</jvmFlag>
            </jvmFlags>
            <ports>
                <port>${CONFIG_PORT}</port>
            </ports>
            <labels>
                <key1>${project.version}</key1>
            </labels>
        </container>
    </configuration>
</plugin>

Prepare Kubernetes manifest files

Prepare Kubernetes manifest files using the supplied bash script:

# cd to kubernetes folder
cd ../kubernetes
source ../.scripts/prepare-kubernetes-manifest-files.sh

Create Secrets in Kubernetes

You can create Secrets for deploying micro service apps in Kubernetes:

kubectl apply -f deploy/0-secrets.yaml

# you can view Secrets in Kubernetes using:
kubectl get secret piggymetrics -o yaml

Deploy Spring Cloud Config Server

You can deploy the Spring Cloud Config Server to Kubernetes:

kubectl apply -f deploy/1-config.yaml

Deploy Spring Cloud Service Registry

You can deploy the Spring Cloud Service Registry to Kubernetes:

kubectl apply -f deploy/2-registry.yaml

You can validate that a Spring Cloud Config Server is up and running by invoking its REST API.

The Spring Cloud Config Server REST API has resources in the following form:

/{application}/{profile}[/{label}]
/{application}-{profile}.yml
/{label}/{application}-{profile}.yml
/{application}-{profile}.properties
/{label}/{application}-{profile}.properties

You can get IP addresses of Spring Cloud Config Server and Spring Cloud Service Registry using kubectl:

kubectl get services
NAME         TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)          AGE
config       LoadBalancer   10.0.34.171   51.143.50.77    8888:31892/TCP   3m15s
kubernetes   ClusterIP      10.0.0.1      <none>          443/TCP          3h42m
registry     LoadBalancer   10.0.198.13   52.137.96.211   8761:31689/TCP   6s

Try:

open http://<EXTERNAL-IP-of-config>:8888/gateway/profile
open http://<EXTERNAL-IP-of-config>:8888/account-service/profile
open http://<EXTERNAL-IP-of-config>:8888/statistics-service/profile
open http://<EXTERNAL-IP-of-config>:8888/notification-service/profile
...
open http://<EXTERNAL-IP-of-config>:8888/notification-service/profile/development
...

You can validate that a Spring Cloud Service Registry is up and running by opening the Service Registry Dashboard:

open http://<EXTERNAL-IP-of-registry>:8761/

Deploy Spring Cloud Gateway

You can deploy the Spring Cloud Gateway to Kubernetes:

kubectl apply -f deploy/3-gateway.yaml

Deploy 4 Spring Cloud micro service apps

You can deploy Spring Cloud micro service apps to Kubernetes:

kubectl apply -f deploy/4-auth-service.yaml
kubectl apply -f deploy/5-account-service.yaml
kubectl apply -f deploy/6-statistics-service.yaml
kubectl apply -f deploy/7-notification-service.yaml

You can validate that Spring Cloud middleware components and micro service apps are running:

kubectl get services
NAME                   TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)          AGE
account-service        ClusterIP      10.0.215.174   <none>          6000/TCP         108s
auth-service           ClusterIP      10.0.164.45    <none>          5000/TCP         2m26s
config                 LoadBalancer   10.0.34.171    51.143.50.77    8888:31892/TCP   24m
gateway                LoadBalancer   10.0.154.13    52.143.88.12    80:31412/TCP     4m3s
kubernetes             ClusterIP      10.0.0.1       <none>          443/TCP          4h3m
notification-service   ClusterIP      10.0.169.231   <none>          8000/TCP         63s
registry               LoadBalancer   10.0.198.13    52.137.96.211   8761:31689/TCP   21m
statistics-service     ClusterIP      10.0.217.229   <none>          7000/TCP         76s

You can also validate that by opening the Spring Cloud Service Registry Dashboard

open http://<EXTERNAL-IP-of-registry>:8761/

Open Spring Cloud micro service apps running on Kubernetes

Open the Piggymetrics landing page by using thegateway app's EXTERNAL-IP.

open http://<EXTERNAL-IP-of-gateway>/

For example:

Troubleshooting micro service apps in Azure Kubernetes Service

With out-of-the-box support for aggregating logs, metrics, and distributed app traces into Azure Monitor, you can easily visualize how your applications are performing, detect and diagnose issues across micro service applications and their dependencies, drill into monitoring data for troubleshooting and gain better understanding of what end-users do with your apps.

Debug in development machine

You can run Spring Cloud Config, Spring Cloud Service Registry, Spring Cloud Gateway and other Spring Cloud components on their dev machine. You can attach debuggers to Spring Cloud micro service apps and step through them. You can look at logs and metrics. Use Java Flight Recorder, etc.

Stream logs from micro service apps in cloud to development machines

You can stream logs from micro service apps running on Kubernetes to your development machine using kubectl, like:

# Stream logs from Spring Cloud Config Server
kubectl logs -f --timestamps=true -l app=config

# Stream logs from Spring Cloud Service Registry
kubectl logs -f --timestamps=true -l app=registry

# Stream logs from Spring Cloud Gateway
kubectl logs -f --timestamps=true -l app=gateway

# Stream logs from Spring Cloud micro service apps
kubectl logs -f --timestamps=true -l app=auth-service
kubectl logs -f --timestamps=true -l app=account-service
kubectl logs -f --timestamps=true -l app=statistics-service
kubectl logs -f --timestamps=true -l app=notification-service

Use aggregated logs and metrics in Azure Log Analytics

You can aggregate logs in Azure Log Analytics and retrieve them using Kusto queries. If you do not have a Log Analytics Workspace in Azure, see how to create a Log Analytics Workspace

You can onboard your Kubernetes cluster to Azure Monitor for monitoring, by clicking on the Logs blade in the Azure Portal and choosing your Log Analytics Workspace:

Then, you can view logs using Kusto queries in the Logs blade of your Log Analytics Workspace:

Here are some sample Kusto queries for viewing logs for each of the micro service apps - please replace java-on-aks with your Azure Kubernetes cluster name:

-- Logs for Spring Cloud Config Server
let ContainerIdList = KubePodInventory
| where ContainerName contains 'config'
| where ClusterId contains 'java-on-aks'
| distinct ContainerID;
ContainerLog
| where ContainerID in (ContainerIdList)
| where LogEntry !contains "AI:" 
| project LogEntrySource, LogEntry, TimeGenerated, Computer, Image, Name, ContainerID
| order by TimeGenerated desc
| render table


-- Logs for Spring Cloud Service Registry
let ContainerIdList = KubePodInventory
| where ContainerName contains 'registry'
| where ClusterId contains 'java-on-aks'
| distinct ContainerID;
ContainerLog
| where ContainerID in (ContainerIdList)
| where LogEntry !contains "AI:" 
| project LogEntrySource, LogEntry, TimeGenerated, Computer, Image, Name, ContainerID
| order by TimeGenerated desc
| render table


-- Logs for Spring Cloud Gateway
let ContainerIdList = KubePodInventory
| where ContainerName contains 'gateway'
| where ClusterId contains 'java-on-aks'
| distinct ContainerID;
ContainerLog
| where ContainerID in (ContainerIdList)
| where LogEntry !contains "AI:" 
| project LogEntrySource, LogEntry, TimeGenerated, Computer, Image, Name, ContainerID
| order by TimeGenerated desc
| render table


-- Logs for Account Service micro service app
let ContainerIdList = KubePodInventory
| where ContainerName contains 'account-service'
| where ClusterId contains 'java-on-aks'
| distinct ContainerID;
ContainerLog
| where ContainerID in (ContainerIdList)
| where LogEntry !contains "AI:" 
| project LogEntrySource, LogEntry, TimeGenerated, Computer, Image, Name, ContainerID
| order by TimeGenerated desc
| render table


-- Logs for Auth Service micro service app
let ContainerIdList = KubePodInventory
| where ContainerName contains 'auth-service'
| where ClusterId contains 'java-on-aks'
| distinct ContainerID;
ContainerLog
| where ContainerID in (ContainerIdList)
| where LogEntry !contains "AI:" 
| project LogEntrySource, LogEntry, TimeGenerated, Computer, Image, Name, ContainerID
| order by TimeGenerated desc
| render table

Use Application Insights to monitor your micro service apps

You can use Application Insights to monitor your live web application. It will automatically detect performance anomalies. It includes powerful analytics tools to help you diagnose issues and to understand what users actually do with your app. It is designed to help you continuously improve performance and usability.

After some time, you can see distributed tracing in the configured Application Insights instance. Go to the `Application Map' blade in the Azure Portal:

Also, you can view the performance and call drill downs in the Performance blade:

Automate and rapidly deploy changes to Azure Kubernetes Service - GitHub Actions or Azure Pipelines

Create an Azure Pipelines CI/CD pipeline that automatically builds the code and deploys it to the Azure Kubernetes Cluster whenever there's a commit to the repository.

Prerequisites

There are some additional prerequisites for this automation:

• GitHub Account
• Azure DevOps Organization
• Environment with AKS resource which creates a service account in the chosen cluster and namespace, which will be used by Azure DevOps account to deploy to AKS
• Azure service connection using service principal to establish authentication between Azure & Azure DevOps services

Important

To simplify the service connection, use the same email address for Azure DevOps as you use for Azure.

• Create an Azure KeyVault and upload secrets. Ensure the service principal used in the service connection above has GET, LIST permissions on the vault. Use below command for that:

az keyvault set-policy -n $KV_NAME --secret-permissions get list --spn <clientId from the Azure SPN JSON>

Azure Pipelines

1. Sign into Azure Pipelines and Create a Pipeline using the azure-pipelines.yml file.

2. Take a look at the pipeline to see what it does. Make sure that all the default inputs are appropriate for your code.

   The azure-pipelines.yml file contains the following key elements:

   • The trigger at the top indicates the commits that trigger the pipeline, such as commits to the master branch.
   • The variables which parameterize the YAML template
   • The stages
     • Build stage, which builds your app, and a Deploy stage, which deploys it to AKS cluster.
     • Deploy stage also refers the Environment with Kubernetes resource. Ensure to modify the environment name to the one that you have created.
   • AzureKeyVault task is used in both the stages to fetch the secrets from Azure Key Vault instance and set as variables. In the Deploy stage, these variables are used to set secrets in the pods.
   • Kubernetes Manifest task has the added benefits of being able to check for object stability before marking a task as success/failure, perform artifact substitution, add pipeline traceability-related annotations onto deployed objects, simplify creation and referencing of imagePullSecrets

   

3. After you've looked at what the pipeline does, select Save and run, after which you're prompted for a commit message because Azure Pipelines adds the azure-pipelines.yml file to your repository. After editing the message, select Save and run again to see your pipeline in action.

4. As your pipeline runs, watch as your build stage, and then your deployment stage, go from blue (running) to green (completed). You can select the stages and jobs to watch your pipeline in action.

   

5. Additionally, you can also explore Kubernetes objects created and the deployment history for the App by navigating to the "Environment".

   • From the pipeline summary -> Select the Environments tab -> Select View environment
   • Resources view within the environment provides a glimpse of the status of objects within the namespace mapped to the resource. It also overlays pipeline traceability on top of these objects so that one can trace back from a Kubernetes object to the pipeline and then back to the commit.

   

GitHub Actions

You can also configure a Workflow to automate build and deploy in GitHub using the workflow.yml.

• Create secrets for every parameter in .scripts/setup-env-variables-azure-template.sh.

  • Set GitHub Repo Secrets AZURE_CREDENTIALS, AKS_CLUSTER_NAME, AKS_RESOURCE_GROUP and AKS_NAMESPACE with Azure creds, AKS cluster name, resource group name and namespace
  • For configuring App secrets, You could either set them as GitHub Secrets or fetch them from an Azure Key Vault instance.

• workflow.yml contains the following key elements:

  • The on: [push] at the top indicates the commits that trigger the pipeline, such as every commit to the branch.
  • The env: with variables which parameterize the YAML template
  • The jobs : Build job, which builds your app, and a Deploy job, which deploys it to AKS cluster. Each job has steps which either uses an action or run a script, all of which execute on a hosted-runner defined by runs-on.
  • AzureKeyVault action is used in both the jobs to fetch the secrets from Azure Key Vault instance and set as environment variables. In the Deploy job, these variables are used to set secrets in the pods.
  • Actions to deploy to AKS are then used to create imagepullsecret, set secrets on pods and finally to deploy to AKS cluster.

Rapidly deploy changes to Azure Kubernetes Service without disruption - blue-green deployments

• To be supplied

Scale out micro service apps in Azure Kubernetes Service

You can scale out micro service apps in Azure Kubernetes Service:

kubectl scale deployment gateway --replicas=4

kubectl describe deployment gateway
Name:                   gateway
Namespace:              default
CreationTimestamp:      Sat, 07 Dec 2019 22:36:27 -0800
Labels:                 app=gateway
                        project=piggymetrics
                        tier=frontend
Annotations:            deployment.kubernetes.io/revision: 1
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"gateway","project":"piggymetrics","ti...
Selector:               app=gateway,project=piggymetrics,tier=frontend
Replicas:               4 desired | 4 updated | 4 total | 4 available | 0 unavailable

Congratulations and Next Steps to Production

Congratulations!!

You built, deployed, scaled out and setup monitoring for Spring Cloud micro service apps using Spring Boot and Spring Cloud, Azure Kubernetes Service, Azure Container Registry, Azure Monitor, Log Analytics and Application Insights.

To advance Java deployments on Kubernetes to production, you should think about the following next steps:

• Frame a scalable infrastructure with Azure Kubernetes Service and Azure Container Registry to satisfy your deployments' business and technical requirements. See Microservices architecture on Azure Kubernetes Service (AKS)
• Frame a plan to assume infrastructure management -- including load balancers, clustering, security, SSL offloading, domain name mapping, etc.
• Define an application lifecyle for apps on Azure Kubernetes Service and use them across developer and DevOps automation tools. This lifecyle should include a source-to-container strategy
• Frame a plan to assume service management -- including configuration, managing secrets, backup & restore, monitoring & log shipping, upgrades, auto-restart, auto-scale or manual-scale, etc. If you are deploying Spring Cloud micro service apps, service management should include strategies for manually or dynamically scaling Spring Cloud middleware components - Spring Cloud Config Server, Spring Cloud Service Registry, Spring Cloud Gateway, etc.

Resources

• Azure Kubernetes Story
• Azure Container Registry
• Production Ready and Supported Java Container Images
• Kusto Query Language
• kubectl Cheat Sheet 1
• kubectl Cheat Sheet 2
• Debugging Applications in Kubernetes Cluster
• Triage Micro Service Applications using Application Map
• Azure for Java Cloud Developers
• Spring Cloud Azure
• Spring Cloud
• Azure Pipelines - Build and Deploy to Azure Kubernetes Service
• Azure Pipelines - Azure Key Vault Task
• GitHub Actions - Workflow to automate build and deploy
• ...

Credits

This Java micro services sample is forked from sqshq/Piggymetrics - see Piggymetrics README.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.