api-vip / k3s_registration_address

[tuxpeople]

Hi

This topic is about an mutli-master setup with internal etcd.

In my current setup, I use Ansible to to deploy a pair of HAproxy nodes and keepalived for them, and use the keepalived-VIP later for this role as k3s_registration_address and also as a part of tls-san.

I'd like to get rid of them. I already stopped using them for ingress, as ingress is now Traefik with MetalLB. I luse flux to deplay the rest of my cluster, and I also would like to deploy keepalived-ingress-vip (https://github.com/janeczku/keepalived-ingress-vip) or kube-vip there. But I need that already earlier, for k3s_registration_address right?

Can I use k3s_server_manifests_templates or k3s_server_manifests_urls for this? Does it get deployed before the additional nodes join? Or is k3s_registration_address only for the initial join and it will never be used again? If so, I could just use first node's IP for joining and deploy vip later.

Br
Thomas

[xanmanning]

k3s_registration_address is the address that worker nodes use to connect to the control plane on initial join of the cluster, if I recall though, if you restart a worker and it isn't able to connect to it then it does fail to start.

Or at least it did at some point in the past, who knows if it still does?

[tuxpeople]

Is k3s_server_manifests_urls being deployed before other masters joining the cluster? If so, I could use it for kube-vip or similiar and use that IP for k3s_registration_address. Does the role wait a while for k3s_registration_address to become available?

[xanmanning]

Yep, k3s_server_manifests_urls is deployed before other "masters" join - they are actually installed before K3s is even installed:

ansible-role-k3s/tasks/state-installed.yml

Lines 37 to 40 in 0bdf538

	- import_tasks: build/preconfigure-k3s-auto-deploying-manifests.yml
	when:
	- k3s_primary_control_node
	- k3s_server_manifests_templates | length > 0

The role doesn't explicitly wait for k3s_registration_address, it waits for the initial "master" to start and be listening on the right port:
https://github.com/PyratLabs/ansible-role-k3s/blob/main/tasks/validate/state/control-plane.yml

However post install it waits for all the workers to be ready:

ansible-role-k3s/tasks/validate/state/nodes.yml

Lines 3 to 20 in 0bdf538

	- name: Check that all nodes to be ready
	ansible.builtin.command:
	cmd: "{{ k3s_install_dir }}/kubectl get nodes"
	changed_when: false
	failed_when: kubectl_get_nodes_result.stdout.find("was refused") != -1 or
	kubectl_get_nodes_result.stdout.find("ServiceUnavailable") != -1
	register: kubectl_get_nodes_result
	until:
	- kubectl_get_nodes_result.rc == 0
	- kubectl_get_nodes_result.stdout.find("NotReady") == -1
	retries: 30
	delay: 20
	when:
	- k3s_control_node
	- ("flannel-backend" not in k3s_runtime_config
	or k3s_runtime_config["flannel-backend"] != "none")
	- not ansible_check_mode
	become: "{{ k3s_become_for_kubectl | ternary(true, false, k3s_become_for_all) }}"