cluster-api-provider-proxmox (CAPPX)

cluster-api-provider-proxmox is a Cluster API infrastructure provider implementation for Proxmox VE.

Description

cluster-api-provider-proxmox provides only infrastructure controller (ProxmoxCluster and ProxmoxMachine). To bootstrap your cluster/machine you need to provide Control Plane provider and Bootstrap provider. For example KubeadmControlPlane and KubeadmBootstrap.

Quick Start

You’ll need a Kubernetes cluster to run against. You can use KIND to get a local cluster for testing, or run against a remote cluster. Note: Your controller will automatically use the current context in your kubeconfig file (i.e. whatever cluster kubectl cluster-info shows).

with KubeadmControlplane/Bootstrap

1. Initialize Management cluster

for more information : https://cluster-api.sigs.k8s.io/user/quick-start.html#initialize-the-management-cluster

# install cluster-api components
export EXP_CLUSTER_RESOURCE_SET=true
clusterctl init --infrastructure=proxmox:v0.3.5 --config https://raw.githubusercontent.com/k8s-proxmox/cluster-api-provider-proxmox/main/clusterctl.yaml

Note: container images are available at ghcr.io/k8s-proxmox/cluster-api-provider-proxmox:<tag>

2. Create your first workload cluster

# export env variables
export CONTROLPLANE_HOST=X.X.X.X                   # control-plane vip
export PROXMOX_URL=https://X.X.X.X:8006/api2/json
export PROXMOX_PASSWORD=password
export PROXMOX_USER=user@pam

# generate manifests (available flags: --target-namespace, --kubernetes-version, --control-plane-machine-count, --worker-machine-count)
clusterctl generate cluster cappx-test --control-plane-machine-count=3 --infrastructure=proxmox:v0.3.5 --config https://raw.githubusercontent.com/k8s-proxmox/cluster-api-provider-proxmox/main/clusterctl.yaml > cappx-test.yaml

# inspect and edit
vi cappx-test.yaml

# apply manifests
kubectl apply -f cappx-test.yaml

3. Access your first workload cluster !!

Usually it takes 2~10 mins to complete bootstrapping the nodes.

# get workload cluster's kubeconfig
clusterctl get kubeconfig cappx-test > kubeconfig.yaml

# get node command for workload cluster
kubectl --kubeconfig=kubeconfig.yaml get node
### example output: this is your first workload cluster !!
## NAME                            STATUS     ROLES           AGE     VERSION
## cappx-test-controlplane-qc9vw   NotReady   control-plane   6m53s   v1.27.3

3-a. [OPTIONAL] Apply your favorite CNI to your workload cluster

Until you apply CNI to your cluster, all the node is NotReady. After this step, all your nodes will become Ready :)

# use weave-cni for this example
kubectl --kubeconfig=kubeconfig.yaml apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml

4. Tear down your workload cluster

kubectl delete cluster cappx-test

Fetures

• No need to prepare vm templates. You can specify any vm image in ProxmoxMachine.Spec.Image. CAPPX bootstrap your vm from scratch. (Supports iso type of image format.)

• Supports custom cloud-config (user data). CAPPX uses VNC websockert for bootstrapping nodes so it can applies custom cloud-config that can not be achieved by only Proxmox API.

• Flexible vmid/node assigning. You can flexibly assign vmid to your qemu and flexibly schedule qemus to proxmox nodes. For more details please check qemu-scheduler.

Node Images

CAPPX is compatible with iso format of image. You can build your own node image and use it for ProxmoxMachine.

CAPPX relies on a few prerequisites which have to be already installed in the used operating system images, e.g. a container runtime, kubelet, kubeadm,.. .

To build your custom node image, you can use kubernetes-sigs/image-builder project.

If it isn't possible to pre-install those prerequisites in the image, you can always deploy and execute some custom scripts through the ProxmoxMachine.spec.cloudInit or KubeadmConfig. Example MD can be found ubuntu2204.yaml.

Compatibility

Proxmox-VE REST API

CAPPX is tested with pve-manager/7.4-3/9002ab8a (running kernel: 5.15.102-1-pve).

Cluster API

	Cluster API v1alpha4	Cluster API v1beta1
CAPPX v1beta1 (v0.x)	?	✓

ControlPlane & Bootstrap provider

CAPPX is tested with KubeadmControlPlane and KubeadmBootstrap.

How it works

This project aims to follow the Cluster API Provider contract.

ProxmoxCluster

Because Proxmox-VE does not provide LBaaS solution, CAPPX does not follow the typical infra-cluster logic. ProxmoxCluster controller reconciles only Proxmox storages used for instances. You need to prepare control plane load balancer by yourself if you creates HA control plane workload cluster. In the cluster-template.yaml, you can find HA control plane example with kube-vip.

ProxmoxMachine

ProxmoxMachine controller follows the typical infra-machine logic. To bootstrap your machine, CAPPX supports only cloud-config type bootstrap data secret. CAPPX is mainly tested with KubeadmControlPlane and KubeadmBootstrap.

Development

Testing

Unit Testing

make unit-test

Unit and Integration Testing

export PROXMOX_URL=https://X.X.X.X:8006/api2/json
export PROXMOX_PASSWORD=password
export PROXMOX_USER=user@pam

make test

E2E Testing

export CONTROLPLANE_HOST=X.X.X.X
export PROXMOX_URL=https://X.X.X.X:8006/api2/json
export PROXMOX_PASSWORD=password
export PROXMOX_USER=user@pam

make e2e

Contributing

Are you interested in contributing to cluster-api-provider-proxmox? Do not hesitate to open GitHub issues.

License

Copyright 2023 Teppei Sudo.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.