Distributed computation with GridGain and Mule 4 on Kubernetes

In this example, we are going to create a system to collect quizzes. We'll deploy it as a set of clustered micro-services on Kubernetes to ensure high availability, we'll use GridGain to support distributed computation, and manage the entire spring-boot system with Spring Boot Admin Server.

Table of Contents

• Getting Started
  • Overview
    • Architecture
    • Data acquisition phase
    • Data processing phase
  • Prerequisites
    • Preparing the environment
    • Required system resources
  • Kubernetes deployment, putting it all together
  • Monitoring instances
  • Testing this example
    • Play with Kubernetes scaling
• Appendix
  • Test your own modifications
  • Full clean-up this sample from your computer
  • Learn more
• License

Getting Started

Overview

The goal of this example is to show you how to join Mule's productivity with the Spring Boot ecosystem using the Spring Boot Starter for Mule 4. Gain high availability deploying Mule Applications as clustered micro-services on Kubernetes and perform distributed data processing using the GridGain Connector for Mule 4.

Architecture

Our system to collect quizzes is composed of:

• A REST Api: It's a Mule application deployed as a clustered micro-service (data acquisition phase and other operations like queries over distributed database).
• A Worker: It's a Mule application deployed as a clustered micro-service for distributed data processing (data processing phase).
• An GridGain Server cluster with persistence enabled for distributed data storage and distributed computation.
• A Spring Boot Admin Server cluster to monitor our Spring Boot applications.

Data acquisition phase

1. User's requests are balanced by internal Kubernetes load balancer between our REST Api nodes to ensure high availability.

2. Every received Quiz (POST request) by our REST API will be enqueued into an GridGain Queue to perform distributed data processing by Workers.

Data processing phase

Workers will listen on a distributed Queue for new Quizzes to process:

• Avoiding more than one Quiz per surveyed (distributed LOCK scope).
• Updating Quizzes with additional data.
• Storing Quizzes into GridGain's distributed database (SQL Query Entity).
• Updating global stats.

Prerequisites

• A local kubernetes development environment
  • JDK 8+
  • Maven 3.5.2+
  • Git
  • Docker Desktop, with enabled Kubernetes
  • Kubernetes ingress
  • Kubernetes dashboard (optional)
• Optionally, Anypoint Studio 7.5+ to work with Mule Applications.

Preparing the environment

Before you start deploying, you need to observe the following steps:

• Create this directory on the root of your file system to store GridGain Data: /opt/k8s/ignite-work (on windows c:\opt\k8s\ignite-work)
• Create this directory on the root of your file system to share some host artifacts: /opt/k8s/shared (on windows c:\opt\k8s\shared)
• Add aliases to localhost for Kubernetes ingresses virtual hosts. Edit the hosts file on your machine and add the aliases mule-api.local, sbadmin.local and ignite-rest.local to the localhost's IP:

  127.0.0.1 localhost mule-api.local sbadmin.local ignite-rest.local
  

  • /etc/hosts on linux/mac
  • c:\windows\system32\drivers\etc\hosts on windows

Required system resources

• Ensure Kubernetes has enough system resources to run this example:

  Required resources:

  • At least 6 GB free RAM space on reserved memory for Kubernetes.
  • At least 4 GB free DISK space on local file system.

  Run kubectl describe nodes to ensure you have enough available resources:

  ...
  Allocated resources:
    (Total limits may be over 100 percent, i.e., overcommitted.)
    Resource           Requests    Limits
    --------           --------    ------
    cpu                750m (9%)   0 (0%)
    memory             140Mi (2%)  340Mi (7%)
    ephemeral-storage  0 (0%)      0 (0%)
  Events:              <none>

  Increase assigned Memory if necessary, open Docker Desktop Preferences -> Resources -> Advanced:

  

• Allow Docker Desktop access to local file system, open Docker Desktop Preferences -> Resources -> File sharing:

  NOTE: On Docker Desktop for Windows select unit C

  

Kubernetes deployment, putting it all together

Let's start by building the required artifacts.

1. Clone this project:

   git clone https://github.com/hawkore/examples-spring-boot-mule4-kubernetes.git
   cd examples-spring-boot-mule4-kubernetes

2. Ensure Docker and Kubernetes are started and ready.

3. Build the required artifacts at once (Mule Applications and Docker images):

   mvn clean install -Pdocker -Dmule.bom.version=4.2.2-hf5

4. Copy packaged Mule applications into shared folder:

   • mule-api-app/target/mule-api-app-1.0.0-mule-application.jar into /opt/k8s/shared directory (on windows c:\opt\k8s\shared).
   • mule-worker-app/target/mule-worker-app-1.0.0-mule-application.jar into /opt/k8s/shared directory (on windows c:\opt\k8s\shared).

Follow by deploying on Kubernetes. Please, observe the following steps in order within examples-spring-boot-mule4-kubernetes directory:

1. Create mandatory Kubernetes artifacts

   kubectl apply -f kubernetes/1-mandatory.yaml

2. Create persistent volumes for your Operating System:

   • For Linux/Mac: Apply this k8s yaml file for persistent volumes on Linux/Mac

     NOTE: Requires that /opt/k8s/ignite-work and /opt/k8s/shared directories exist

     kubectl apply -f kubernetes/2-local-disk-volumes-linux-mac.yaml

   • For Windows: Apply this k8s yaml file for persistent volumes on Windows:

     NOTE: Requires that c:\opt\k8s\ignite-work and c:\opt\k8s\shared directories exist

     kubectl apply -f kubernetes/2-local-disk-volumes-windows.yaml

3. Deploy Spring Boot Admin Server on Kubernetes (2 replicas):

   kubectl apply -f kubernetes/3-statefulset-sb-admin-server.yaml

4. Deploy Spring Boot GridGain Server on Kubernetes (1 replica):

   kubectl apply -f kubernetes/4-statefulset-ignite-server-node.yaml

5. Once Spring Boot GridGain Server is running, install and activate a license for GridGain Connector for Mule 4

6. Deploy Quiz REST Api - Mule Application on Kubernetes (1 replica):

   kubectl apply -f kubernetes/6-statefulset-mule-api-app.yaml

7. Deploy Worker - Mule Application on Kubernetes (2 replicas):

   kubectl apply -f kubernetes/7-statefulset-mule-worker-app.yaml

Monitoring instances

Open your browser and play with Spring Boot Admin Server:

Testing this example

Open your browser and play with Quiz REST Api console:

Play with Kubernetes scaling

1. Scale mule-worker-app to 1 replica and see what happens on Spring Boot Admin Server Wallboard:

   kubectl scale -n my-mule4-stack statefulset mule-worker-app --replicas=1

2. Stop all Workers (scale to 0) and see what happens when you create new Quizzes (POST request) and GET stats from Quiz REST Api console

   totalReceived should be increased on every new received Quiz (POST request); totalProcessed and totalDuplicated should remain un-altered

   kubectl scale -n my-mule4-stack statefulset mule-worker-app --replicas=0

   

3. Scale mule-worker-app to 2 replicas and see what happens when GET stats from Quiz REST Api console

   Once Workers finish processing enqueued Quizzes, totalProcessed + totalDuplicated should be equals to totalReceived

   kubectl scale -n my-mule4-stack statefulset mule-worker-app --replicas=2

   

4. List Quizzes to see how data processing was distributed between nodes:

   

   • Quiz's data:
     • id: Unique identifier (quizCache key).
     • email: Surveyed email.
     • yes: a YES response.
     • no: a NO response.
     • na: a Non Answered Quiz.
     • qts: The timestamp when a Quiz was received by an Api Node.
     • pts: The timestamp when a Quiz was processed by a Worker node.
     • apiIp: the Api Node IP that received the Quiz.
     • workerIp: the Worker Node IP that processed the Quiz.

Appendix

Test your own modifications

If you want to modify mule-api-app or mule-worker-app and re-deploy them, follow these steps:

1. Package again and copy the generated jar file into /opt/k8s/shared directory, for example:

   cd mule-api-app
   mvn clean package
   cp target/mule-api-app-1.0.0-mule-application.jar /opt/k8s/shared/

2. Restart the Kubernetes statefulset of your modified Mule application, for example:

   kubectl rollout restart -n my-mule4-stack statefulset/mule-api-app

Clean-up

To full clean-up this sample from your computer, follow these steps:

1. Remove namespace from Kubernetes:

   kubectl delete namespace my-mule4-stack

2. Remove persistent volumes:

   kubectl delete persistentvolume ignite-storage
   kubectl delete persistentvolume shared-storage

3. Optionally, delete /opt/k8s directory. WARNING: If you delete /opt/k8s directory you will lose the license activation for the GridGain Connector!!

Learn more

• About Mule ESB.
• About GridGain.
• About Spring Boot Starter for Mule 4.
• About Spring Boot Admin Server.

License

Copyright 2020 HAWKORE, S.L.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

About subcomponents

This project may contain subcomponents with separate copyright notices and license terms. Your use of the source code for these subcomponents is subject to their terms and conditions.