This repo contains all the required manifests and documentation to build a modern observability stack built on opensource software components. This implementation utilizes a Linux Virtual Machine as a Bastion Host to prepare a devops environment with all the necessary tools to required to deploy the pre-configured pre-plumbed observability stack which is comprised of:
- Logstash
- ElasticSearch
- Kibana
- AlertManager
- Prometheus
- Grafana
- NodeExporter
This implementation utilizes F5 BigIP Virtual Instances deployed as an auto-scaleset to provide ingress services and ITSG compliant security controls for the solution. You will require between 2 and 5 HP-VE BEST Bundle License keys with IP Intelligence and Threat Campaigns add-ons. Contact you local F5 team for evaluation licenses if you want to kick the tires on this without any financial commmitment.
NB: Autoscaling is not enabled by default. Manual scaling of both the F5 BIG-IP VMSS and the AKS deployments is left up to the user. This example supports up to a maximum of 5 instances in the VMSS.
This solution was designed an implemented Government of Canada federal clients in mind and all the constraints and regulatory compliance that comes with that type of deployment pattern. These constraints force certain design choices that might otherwise not be optimal.
For extended details on the solution design please refer to the DESIGN.md page.
This implementation is designed to deploy into an existing Azure resource-group and properly configured virtual network. If required, use these steps to build these dependencies using the method of your choice:
- Clone the repo to your local environment.
- Create a Resource Group in the Azure region of your choice.
- Create 3 Network Security Groups:
- mgmtNsg (default ruleset)
- extNsg (default ruleset)
- intNsg (default ruleset)
- Create a NAT Gateway & Public IP address for it.
- default: Overwatch-natgw-pip
- default: Overwatch-natgw
- Create a VNet with a large address space (/12).
- The default name is overwatch-vnet
- The default CIDR is 10.120.0.0/12
- Create 3 Subnets for the F5 BigIP VMSS cluster, associating the corresponding NSG created in Step 3:
- mgmt-subnet (default: 10.127.254.0/24)
- associate the Overwatch-natgw created in step 4 with the mgmt-subnet
- associate the mgmtNsg with this subnet
- internal-subnet (default: 10.127.253.0/24)
- associate the intNsg with this subnet
- external-subnet (default: 10.127.252.0/24)
- associate the extNsg with this subnet
- mgmt-subnet (default: 10.127.254.0/24)
- From the Azure Marketplace, create a new Linux Virtual Machine using the Ubuntu 22.04-LTS:
- Create a new resource
- Search: Ubuntu Server 22.04 LTS
- On the first search result, click Create
Setting Value Resource group Use RG created in Step 2 Virtual machine name tux Region Canada Central / Canada East Availability Zone Zone 1 Security type Trusted launch virtual machines Image Ubuntu Server 22.04 LTS - x64 Gen2 Size Standard_D4s_v3 - 4 vcpus, 16 GiB memory Authentication Type SSH public key / password (your choice) Username azops SSH Public key source Generate new key pair SSH Key Type RSA SSH Format Key Pair name tux Authentication Type Password (your choice) Username azops Password Default1235! Confirm Password Default123345! Public Inbound ports none - At the top of the form, click on Networking
Setting Value Virtual Network Overwatch-vnet Subnet mgmt-subnet Public IP (new) NIC network security group none Enable accelerated networking true Load balancing options none - Click Review + Create NB: Once the resource is created, select it and then start the virtual machine
- While the VM is booting, edit the mgmtNSG and add a rule to allow inbound SSH connections from your workstation:
- Inside the resource group created in Step 2, Click on mgmtNSG
- Click Settings -> Inbound security rules -> Add
Setting Value Source My IP Address Destination Any Service SSH Action Allow - Click: Add
- If using windows, install Windows Terminal along with OpenSSH client (available on the Windows Store) on your local workstation.
- Launch a Terminal session, and SSH to the Public IP address of the Bastion Host using the credentials settings supplied in Step 7.
- Update the BaseOS and install our DevOps tools:
- Copy and Paste the commands as show in the repo artifact: https://raw.githubusercontent.com/f5devcentral/overwatch/refs/heads/main/build_ubuntu_vms.sh
- These commands will perform the following essential tasks:
- Install all OS updates and security patches
- Install cli tools: curl, wget, net-tools, python3-pip, ansible, gnupg, git, jq
- Install devops tools:
- Terraform
- kubectl
- k9s
- Azure CLI
- Filebeat agent NB: You will most likely want to reboot the Bastion Host Virtual Machine for any kernel updates to take effect.
- Clone this Repo to your Bastion Host:
mkdir -p ~/code && cd ~/code && git clone https://github.com/f5devcentral/overwatch.git- Customize the default values for the F5 BigIP VMSS configuration
cd ~/code/overwatch/terraform/az-auto-scaleset
cp terraform.tfvars.example terraform.tfvars
vi/nano terraform.tfvarsNB: ***CRITICAL*** Set Custom Username and Password and SSH Public Key location values at a minimum.
NB: Customize other values to match your Azure environment.
- Customize the default values for the aks-cluster deployment:
cd ~/code/overwatch/terraform/az-aks-cluster
cp terraform.tfvars.example terraform.tfvars
vi terraform.tfvarsNB: ***CRITICAL*** Set Custom Username and Password values at a minimum.
- Configure CIS with the same credentials configured in Step 13.
cd ~/code/overwatch/helm/cis
vi cis.shNB: ***CRITICAL*** set the user and pass values to match those used when you deployed the F5 BigIP VMSS in Step 15.
- Deploy the F5 BigIP VMSS:
cd ~/code/overwatch/terraform/az-auto-scaleset
terraform init
terraform plan
terraform apply --auto-approve- Deploy the AKS Cluster:
cd ~/code/overwatch/terraform/az-aks-cluster
terraform init
terraform plan
terraform apply --auto-approve- Deploy F5 Container Ingress Services operator:
cd ~/code/overwatch/helm/cis
./cis.sh- Configure kubectl and helm to remotely administer AKS cluster:
cd ~/code/overwatch
vi/nano kubeconfig.shSet 'rg' value to match the name of the Resource Group created in Step 2.
Set 'aks' value to match the name of the AKS cluster created in step 17.
./kubeconfig.sh NB: You will be prompted to autenticate to the Azure Portal to fetch the credentials needed to remotely mange the AKS cluster
NB: You may be prompted to accept the Terms and Conditions of the F5 BigIP Azure Marketplace image. This is normal and doesn't have a cost associated with it since we're using a BYOL image type
alias k=kubectl
source <(kubectl completion bash)
k get pods -A #you should not see any errorsPart V: Deploy the modern observability software stack 20. Deploy ELK Stack:
cd ~/code/overwatch/helm/eck-stack
vi values.yaml #(adjust as needed or leave defaults if not sure)
./eck.sh- Deploy ELK Stack:
cd ~/code/overwatch/helm/prom-graf-stack
vi values.yaml #(adjust as needed or leave defaults if not sure)
./promgraf.sh- Deploy Ingress Custom Resource Defnitions:
cd ~/code/overwatch/helm/cis/crds
#ECK Ingress
for i in `ls -1 *.yaml`; do (kubectl apply -n elastic-system -f $i)
#Grafana Ingress
for i in `ls -1 *.yml`; do (kubectl apply -n grafana-system -f $i)- Extract the ElasticSearch admin password using the following command:
#Fetch ES Creds
kibanaUrl=$(kubectl get service elastic-es-http)
esPass=`kubectl get secret elasticsearch-es-elastic-user -n elastic-system -o jsonpath="{.data.elastic}" |base64 -d`
echo "URL: ${kibanaUrl}"
echo "Username: elastic"
echo "Password: ${esPass}"
echo ""- Extract the Kibana Login URL:
# show grfana admin creds
grafanaUrl=$(kubectl get service grafana-labs-stack)
echo "Username: admin"
echo "Password: `kubectl get secret grafana -n elastic-system -o jsonpath="{.data.admin-password}" |base64 -d`"
echo ""- Import the Kibana Dashboards - Login to ElasticSearch with credentials retrieved in Step 23. - In the Main Menu, select Stack Management - In the new menu, under Kibana, select Data Management - Select Dashboards - Select Import - Browse to the local copy of the kibana dashboards and select the first bundle (.ndjson) - Repeat for the second and third bundle
The solution includes some pre-fabricated confgurations:
- Logstash Input pipelines:
- Syslog
- 7514 UDP
- 7514 TCP
- f5BigIpAWAF
- 8514 TCP
- HEC
- 7080 HTTP
- Beats
- 5044 TCP
- Syslog
- Prometheus Scrape Configs:
- f5BigIp
- By default Prometheus is configured to scrape Kubernetes metrics along with the F5 BigIP telemetry available from the VMSS cluster providing ingress services to the AKS cluster.
- f5BigIp
For support, please open a GitHub issue. Note, the code in this repository is community supported and is not supported by F5 Networks. For a complete list of supported projects please reference SUPPORT.md.
Please refer to the F5 DevCentral Community Code of Conduct.
Copyright 2014-2020 F5 Networks Inc.
Before you start contributing to any project sponsored by F5 Networks, Inc. (F5) on GitHub, you will need to sign a Contributor License Agreement (CLA).
If you are signing as an individual, we recommend that you talk to your employer (if applicable) before signing the CLA since some employment agreements may have restrictions on your contributions to other projects. Otherwise by submitting a CLA you represent that you are legally entitled to grant the licenses recited therein.
If your employer has rights to intellectual property that you create, such as your contributions, you represent that you have received permission to make contributions on behalf of that employer, that your employer has waived such rights for your contributions, or that your employer has executed a separate CLA with F5.
If you are signing on behalf of a company, you represent that you are legally entitled to grant the license recited therein. You represent further that each employee of the entity that submits contributions is authorized to submit such contributions on behalf of the entity pursuant to the CLA.