How to use a Private Cluster in Kubernetes Engine

Table of Contents

• Introduction
  • Public Clusters
  • Private Clusters
  • Workload Identity Overview
• Demo Architecture
  • Bastion Host
  • Workload Identity
• Prerequisites
  • Cloud Project
  • Required GCP APIs
  • Run Demo in a Google Cloud Shell
    • Install Terraform
    • Install Cloud SDK
    • Install kubectl CLI
  • Authenticate gcloud
  • Configure gcloud settings
• Create Resources
• Validation
• Tear Down
• Troubleshooting
• Relevant Material

Introduction

This guide demonstrates creating a Kubernetes private cluster in Google Kubernetes Engine (GKE) running a sample Kubernetes workload that connects to a Cloud SQL instance using the cloud-sql-proxy "sidecar". In addition, the Workload Identity (currently in Beta) feature is used to provide credentials directly to the cloud-sql-proxy container to facilitate secure tunneling to the cloud sql instance without having to handle GCP credentials manually.

Public Clusters

By default, GKE clusters are created with a public IP address in front of the Kubernetes API (aka "masters" or "the control plane"). In addition, the GCE instances that serve as the worker nodes are given both private and ephemeral public IP addresses. This facilitates ease of administration for using tools like kubectl to access the Kubernetes API and SSH to access the GCE instances for troubleshooting purposes. Assuming the GKE cluster was created on a subnet in the default VPC network of a project, the default access control allows "any" or 0.0.0.0/0 to reach the Kubernetes API and the default firewall rules allow "any" or 0.0.0.0/0 to reach the worker nodes via SSH. These clusters are commonly referred to as "public clusters".

While the authentication and authorization mechanisms for accessing the Kubernetes API over TLS and worker nodes via SSH offers strong protection against unauthorized access, it is strongly recommended that additional steps are taken to limit the scope of potential access:

1. Restrict to a known list of source subnets for access to the Kubernetes cluster API via TLS (tcp/443) using the master_authorized_networks access list configuration setting.
2. Restrict to a known list of source subnets or remove the default firewall rules allowing SSH (tcp/22) to the worker nodes.

This provides several key benefits from a defense-in-depth perspective:

1. Reducing the scope of source IPs that can potentially perform a Denial of Service or exploit against the Kubernetes API server or the SSH daemon running on the worker nodes.
2. Reducing the scope of source IPs that can leverage credentials stolen from a developer laptop compromise, credentials found in source code repositories, or credentials/tokens obtained from resources inside the cluster.
3. Decreasing the likelihood of a newly discovered vulnerability being exploitable and/or granting more time to the team to devise a patching/upgrade strategy.

However, fom an operational perspective, managing these access control lists may not be feasible in every organization. Larger organizations may already have remote access solutions in place either on-premise or in the cloud that they prefer to leverage. They may also have dedicated interconnects which provide direct, high-bandwidth access from their office network to their GCP environments. In these cases, there is no need for the GKE clusters to be publicly accessible.

Private Clusters

GKE offers two configuration items that combine to form what is commonly known as "private clusters":

1. The GCE instances that serve as the Kubernetes worker nodes do not get assigned a public IP. Instead, they are only assigned a private IP from the VPC node subnet. This is the enable_private_nodes configuration setting.
2. The Kubernetes API/GKE API/GKE Control Plane IP is assigned a private IP address from a dedicated subnet for this purpose and is automatically accessible from the node and pod subnets. This is the enable_private_endpoint configuration setting.

When both of these are configured on a GKE cluster, a few key behaviors change:

1. The worker nodes no longer have egress to the Internet, and that will prevent the nodes and pods from having external access. In addition, pods defined using containers from public container image registries like DockerHub will not be able to access and pull those containers. To restore this access, this demo implements a Cloud NAT router to provide egress Network Address Translation functionality.
2. Access to other Google Cloud Platform (GCP) APIs like Google Cloud Storage (GCS) and Google Cloud SQL requires enabling private API access on the VPC Subnet to enable the private routing configuration which directs traffic headed to GCP APIs entirely over the internal GCP network. This demo connects to the Cloud SQL instance via private access.
3. Access to the Kubernetes API/GKE Cluster Control Plane will only be possible from within the VPC subnets. This demo deploys what is known as a bastion host as a dedicated GCE instance in the VPC subnet to allow for an administrator/developer to use SSH Tunneling to support kubectl access.

Workload Identity Overview

The current guide for how to configure the cloud-sql-proxy with the necessary GCP credentials involves creating a service account key in JSON format, storing that in a Kubernetes-native secret inside the namespace where the pod is to run, and configuring the pod to mount that secret on a particular file path inside the pod. However, there are a few downsides to this approach:

1. The credentials inside this JSON file are essentially, static keys and they don't expire unless manually revoked via the GCP APIs.
2. The act of exporting the credential file to JSON means it touches the disk of the administrator and/or CI/CD system.
3. Replacing the credential means re-exporting a new Service Account key to JSON, replacing the contents of the Kubernetes secret with the updated contents, and restarting the pod for the cloud-sql-proxy to make use of the new contents.

Workload Identity helps remove several manual steps and ensures that the cloud-sql-proxy is always using a short-lived credential that auto-rotates on it's own. Workload Identity, when configured inside a GKE cluster, allows for a Kubernetes Service Account (KSA) to be mapped to a GCP Service Account (GSA) via a process called "Federation". It then installs a proxy on each GKE worker node that intercepts all requests to the GCE Metadata API where the dynamic credentials are accessible and returns the current credentials for that GCP Service Account to the process in the pod instead of the credentials normally associated with the underlying GCE Instance. As long as the proper IAM configuration is made to map the KSA to the GSA, the pod can be given a dedicated service account with just the permissions needed.

Demo Architecture

Given a GCP project, the code in this demo will create the following resources via Terraform:

• A new VPC and new VPC subnets
• A Cloud NAT router for egress access from the VPC subnets
• A Cloud SQL Instance with a Private IP
• A GCE Instance serving as a Bastion Host to support SSH Tunneling
• A GKE Cluster running Workload Identity with no public IPs on either the API or the worker nodes
• A sample Kubernetes deployment that uses the cloud-sql-proxy to access the Cloud SQL instance privately and uses Workload Identity to dynamically and securely fetch the GCP credentials.

1. Exposing workloads inside the GKE cluster is done via a standard load balancer strategy.
2. Accessing the Kubernetes API Server/Control Plane from the Internet is through an SSH tunnel on the Bastion Host.
3. GKE worker nodes and pods running on those nodes access the Internet via Cloud NAT through the Cloud Router.
4. GKE worker nodes and pods running on those nodes access other GCP APIs such as Cloud SQL via Private API Access.

Bastion Host

Traditionally, a "jump host" or "bastion host" is a dedicated, hardened, and heavily monitored system that was placed in the DMZ of a network to allow for secure, remote access. In the cloud, this commonly is deployed as a shared instance that multiple users SSH into and work from when accessing cloud resources. Tools like the Google Cloud SDK and Terraform are often installed on these systems. There are two problems with this approach:

1. If users authenticate to GCP using gcloud, their credentials are stored in the /home directory on this instance. The same issue applies to users who obtain a valid kubeconfig file for accessing a GKE cluster.
2. If users log in via SSH and then use su or sudo to switch to a shared account to perform privileged operations, the audit logs will no longer be able to directly identify who performed an action. In the case of sudo to root, that means all GCP credentials in the /home directories are available to be used for impersonation attacks (Alex performs malicious actions with Pat's credentials).

This bastion host attempts to solve for both issues. It runs two services:

1. An OpenSSH) daemon to support SSH access via gcloud compute ssh or via Identity Awareness Proxy.
2. A TinyProxy daemon listening on localhost:8888 that provides a simple HTTP Proxy.

Note: The bastion host is configured to allow SSH access from 0.0.0.0/0 via a dedicated firewall rule, but this can and should be restricted to the list of subnets for your needs.

This means that both gcloud and kubectl commands can still be run on the local developer/administrator workstation, but kubectl commands can be "proxied" through an SSH Tunnel made to the bastion on their way to the Kubernetes API without disrupting the TLS connection and certificate verification process.

From a practical standpoint, using the bastion requires two additional steps for kubectl to reach the private cluster's Kubernetes API IP:

1. Run gcloud compute ssh and forward a local port (8888) to the localhost:8888 on the bastion host where the tinyproxy daemon is listening.
2. Provide an environment variable (HTTPS_PROXY=localhost:8888) when using kubectl to instruct it to use the forwarded port that reaches the tinyproxy daemon running on the bastion host on its way to the Kubernetes API.

Workload Identity

The use case of this demo requires that the pgadmin4 (Postgres SQL Admin UI) container has a cloud-sql-proxy "sidecar" that it uses to connect securely to the Cloud SQL instance. The IAM Role that is needed to make this connection is roles/cloudsql.client. This demo creates a dedicated GCP Service Account, binds the roles/cloudsql.client IAM Role to it at the project level, creates a dedicated Kubernetes Service Account (postgres) in the default namespace, and grants roles/iam.workloadidentityuser on the KSA-to-GSA IAM binding.

The result is that the processes inside pods that use the default/postgres Kubernetes Service Account that reach for the GCE metadata API to retrieve GCP credentials will be given the credentials from the dedicated GCP Service Account with just the Cloud SQL access permissions. There are no static GCP Service Account keys to export, no Kubernetes secrets to manage, and the credentials automatically rotate themselves.

Prerequisites

The steps described in this document require installations of several tools and the proper configuration of authentication to allow them to access your GCP resources.

Cloud Project

If you do not have a Google Cloud account, please signup for a free trial here. You'll need access to a Google Cloud Project with billing enabled. See Creating and Managing Projects for creating a new project. To make cleanup easier it's recommended to create a new project.

Required GCP APIs

The following APIs will be enabled:

• Compute Engine API
• Kubernetes Engine API
• Cloud SQL Admin API
• Secret Token API
• Stackdriver Logging API
• Stackdriver Monitoring API
• IAM Service Account Credentials API

Run Demo in a Google Cloud Shell

Click the button below to run the demo in a Google Cloud Shell.

How to check your account's quota is documented here: quotas.

All the tools for the demo are installed. When using Cloud Shell execute the following command in order to setup gcloud cli. When executing this command please setup your region and zone.

gcloud init

Tools

When not using Cloud Shell, the following tools are required:

• Access to an existing Google Cloud project.
• Bash and common command line tools (Make, etc.)
• Terraform v0.12.3+
• gcloud v255.0.0+
• kubectl that matches the latest generally-available GKE cluster version.

Install Terraform

Terraform is used to automate the manipulation of cloud infrastructure. Its installation instructions are also available online.

Install Cloud SDK

The Google Cloud SDK is used to interact with your GCP resources. Installation instructions for multiple platforms are available online.

Install kubectl CLI

The kubectl CLI is used to interteract with both Kubernetes Engine and Kubernetes in general. Installation instructions for multiple platforms are available online.

Deployment

The steps below will walk you through using Google Kubernetes Engine to create Private Clusters.

Authenticate gcloud

Prior to running this demo, ensure you have authenticated your gcloud client by running the following command:

gcloud auth login

Configure gcloud settings

Run gcloud config list and make sure that compute/zone, compute/region and core/project are populated with values that work for you. You can choose a region and zone near you. You can set their values with the following commands:

# Where the region is us-central1
gcloud config set compute/region us-central1

Updated property [compute/region].

# Where the zone inside the region is us-central1-c
gcloud config set compute/zone us-central1-c

Updated property [compute/zone].

# Where the project name is my-project-name
gcloud config set project my-project-name

Updated property [core/project].

Create Resources

To create the entire environment via Terraform, run the following command:

make create

Apply complete! Resources: 33 added, 0 changed, 0 destroyed.

Outputs:

...snip...
bastion_kubectl = HTTPS_PROXY=localhost:8888 kubectl get pods --all-namespaces
bastion_ssh = gcloud compute ssh private-cluster-bastion --project bgeesaman-gke-demos --zone us-central1-a -- -L8888:127.0.0.1:8888
cluster_ca_certificate = <sensitive>
cluster_endpoint = 172.16.0.18
cluster_name = private-cluster
gcp_serviceaccount = private-cluster-pg-sa@my-project-name.iam.gserviceaccount.com
get_credentials = gcloud container clusters get-credentials --project my-project-name --region us-central1 --internal-ip private-cluster
postgres_connection = my-project-name:us-central1:private-cluster-pg-410120c4
postgres_instance = private-cluster-pg-410120c4
postgres_pass = <sensitive>
postgres_user = postgres
Fetching cluster endpoint and auth data.
kubeconfig entry generated for private-cluster.

Next, review the pgadmin deployment located in the /manifests directory:

cat manifests/pgadmin-deployment.yaml

The manifest contains comments that explain the key features of the deployment configuration. Now, deploy the application via:

make deploy

Detecting SSH Bastion Tunnel/Proxy
Did not detect a running SSH tunnel.  Opening a new one.
Pseudo-terminal will not be allocated because stdin is not a terminal.
SSH Tunnel/Proxy is now running.
Creating the PgAdmin Configmap
configmap/connectionname created
Creating the PgAdmin Console secret
secret/pgadmin-console created
serviceaccount/postgres created
serviceaccount/postgres annotated
Deploying PgAdmin
deployment.apps/pgadmin4-deployment created
Waiting for rollout to complete and pod available.
Waiting for deployment "pgadmin4-deployment" rollout to finish: 0 of 1 updated replicas are available...
deployment "pgadmin4-deployment" successfully rolled out

The make deploy step ran the contents of ./scripts/deploy.sh which did a few things:

1. Created an SSH tunnel to the Bastion Host (if it wasn't running already) that should still be running in the background.
2. Used kubectl to create a configmap containing the connection string for connecting to the correct Cloud SQL Instance, a dedicated service account named postgres in the default namespace, and added a custom annotation to that service account.
3. Ran kubectl to deploy the pgadmin4 deployment manifest.
4. Ran kubectl to wait for that deployment to be up and healthy.

Now, with the SSH tunnel still running in the background, you can interact with the GKE cluster using kubectl. For example:

HTTPS_PROXY=localhost:8888 kubectl get pods --all-namespaces

Because that environment variable must be present for each invocation of kubectl, you can alias that command to reduce the amount of typing needed each time:

alias k="HTTPS_PROXY=localhost:8888 kubectl"

And now, using kubectl looks like the following:

k get pods --all-namespaces
k get namespaces
k get svc --all-namespaces

Note: export-ing the HTTPS_PROXY setting in the current terminal may alter the behavior of other common tools that honor that setting (e.g. curl and other web related tools). The shell alias helps localize the usage to the current invocation of the command only.

Validation

If no errors are displayed during deployment, you should see your Kubernetes Engine cluster in the GCP Console with the sample application deployed. This may take a few minutes.

Validation is fully automated. The validation script checks for the existence of the Postgress DB, Google Kubernetes Engine cluster, and the deployment of pgAdmin. In order to validate that resources are installed and working correctly, run:

make validate

Detecting SSH Bastion Tunnel/Proxy
Detected a running SSH tunnel.  Skipping.
Checking that pgAdmin is deployed on the cluster... pass
Checking that pgAdmin is able to connect to the database instance... pass

The make validate performs two simple checks that can be done manually. Checking the status of the pgadmin4 deployment for health:

k rollout status --timeout=10s -f manifests/pgadmin-deployment.yaml

deployment "pgadmin4-deployment" successfully rolled out

And using kubectl exec to run the pg_isready command from the pgadmin4 container which performs a test connection to the Postgres database and verifies end-to-end success:

k exec -it -n default "$(k get pod -l 'app=pgadmin4' -ojsonpath='{.items[].metadata.name}')" -c pgadmin4 -- pg_isready -h localhost -t 10

localhost:5432 - accepting connections

You may also wish to view the logs of the key pods in this deployment. To see the logs of the pgadmin4 container:

k logs -l 'app=pgadmin4' -c pgadmin4 -f

To see the logs of the cloud-sql-proxy container:

k logs -l 'app=pgadmin4' -c cloudsql-proxy -f 

To see the logs of the gke-metadata-proxy containers which handle requests for "Workload Identity":

k logs -n kube-system -l 'k8s-app=gke-metadata-server' -f

Tear Down

When you are finished with this example you will want to clean up the resources that were created so that you avoid accruing charges. Teardown is fully automated. The destroy script deletes all resources created using Terraform. Terraform variable configuration and state files are also cleaned if Terraform destroy is successful. To delete all created resources in GCP, run:

make teardown

Troubleshooting

• The create script fails with a Permission denied when running Terraform - The credentials that Terraform is using do not provide the necessary permissions to create resources in the selected projects. Ensure that the account listed in gcloud config list has necessary permissions to create resources. If it does, regenerate the application default credentials using gcloud auth application-default login.
• Terraform timeouts - Sometimes resources may take longer than usual to create and Terraform will timeout. The solution is to just run make create again. Terraform should pick up where it left off.

Relevant Material

• Private GKE Clusters
• Workload Identity
• Terraform Google Provider
• Securely Connecting to VM Instances
• Cloud NAT
• Kubernetes Engine - Hardening your cluster's security

Note, This is not an officially supported Google product