Issue: Shrinking a node with k3state uninstalled

[david1412]

Hi Team,

I'm facing a problem when shrinking the node ( scale down) , I used the Roles : control-plane,etcd,master

Problem statement: Scale down node with k3s state uninstalled but the node is not deleting after the successful Ansible job.
Root cause: The task drain and delete the node is working as expected.

Steps i followed:

• To scale down the cluster, I'm using Ansible task k3s_uninstall to shrink the node. In the yml i mentioned k3s_state: uninstalled , so the role uninstalled is executed. ( this role has drain-and-remove node imported, but not working as expected)
• After successful Ansible job, the status of the node is "Not Ready".
  (till this part, this is working fine. Expectation: The scale down node should be remove automatically. )
• There is a task (drain-and-remove nodes) that is imported while k3s_uninstall is ran,The script suppose to do drain and delete the node but the scale down node is not deleted ( drain and remove node not works as expected).

Attached screenshot of the output.
Can anyone suggest, how to fix this issue.
Thanks in Advance :)

[david1412]

Hi @xanmanning , could you please revert to my query. Thanks

[xanmanning]

Hi.

In your hostvars, do you have the nodes you wish to uninstall set to k3s_state: uninstalled and run against the entire cluster?

Effectively all my cluster nodes have k3s_state defined against them:

---

k3s_cluster:
  hosts:
    node1:
      # Some vars
      k3s_state: installed
      k3s_control_node: true
    node2:
      # Some vars
      k3s_state: installed
    node3:
      # Some vars
      k3s_state: uninstalled
      k3s_control_node: true

Works for me.

https://asciinema.org/a/OkABYUy8ZHlRbFaz1iGZmOCBg

[david1412]

Hi
Thanks for your reply.
I tried with my configuration, it didnt work.
Question: in the drain and delete task ,you mentioned kubectl drain version 1.22 to ( we are using 1.21, i changed this part as well) by changing the version , does this task works? the output for this task is skipped!

Could you please guide me on this issue. Thanks

Configuration:

---

inventory.yml

k3s_cluster:
  hosts:
    kube-0:
      ansible_host: ***-npi-**-stage-app-01.example.de
      k3s_control_node: true
      k3s_state: installed
    kube-1:
      ansible_host: ***-npi-***-stage-app-02.example.de
      k3s_state: installed
    kube-2:
      ansible_host: ***-npi-***-stage-app-03.example.de
      k3s_state: installed
    kube-3:
      ansible_host: ***-npi-***-stage-app-04.example.de
      k3s_state: uninstalled ( first install state)

Plan is to drain and remove kube-3 node ( worker). 

kubectl get nodes

NAME                                      STATUS        ROLES                                     AGE   VERSION

***-npi-**-stage-app-01      Ready      control-plane,etcd,master          54m   v1.21.0+k3s1
***-npi-***-stage-app-02    Ready             <none>                                 23m   v1.21.0+k3s1
***-npi-***-stage-app-03    Ready              <none>                                54m   v1.21.0+k3s1
***-npi-***-stage-app-04    NotReady        <none>                                22m   v1.21.0+k3s1

cluster.yml

k3s_build_cluster: true
k3s_etcd_datastore: true
k3s_release_version: v1.21.0+k3s1
k3s_offline_preparation: false
k3s_use_unsupported_config: true
k3s_become: true
k3s_hash_sum: *******
k3s_images_hash_sum: ******
k3s_server:
****
k3s_agent:
 *****

release version: 2.6.0

output:


TASK [ods/k3s_install_v3.1.1 : Ensure uninstalled nodes are drained] *****************************************************************************************************************************************************************************************
Tuesday 03 May 2022  06:52:11 +0000 (0:00:00.667)       0:01:25.740 ***********
skipping: [kube-3] => (item=kube-0)
skipping: [kube-3] => (item=kube-1)
skipping: [kube-3] => (item=kube-2)
skipping: [kube-3] => (item=kube-3)

TASK [ods/k3s_install_v3.1.1 : Ensure uninstalled nodes are removed] *****************************************************************************************************************************************************************************************
Tuesday 03 May 2022  06:52:11 +0000 (0:00:00.239)       0:01:25.980 ***********
skipping: [kube-3] => (item=kube-0)
skipping: [kube-3] => (item=kube-1)
skipping: [kube-3] => (item=kube-2)
skipping: [kube-3] => (item=kube-3)

[david1412]

Hi.

In your hostvars, do you have the nodes you wish to uninstall set to k3s_state: uninstalled and run against the entire cluster?

Effectively all my cluster nodes have k3s_state defined against them:

---

k3s_cluster:
  hosts:
    node1:
      # Some vars
      k3s_state: installed
      k3s_control_node: true
    node2:
      # Some vars
      k3s_state: installed
    node3:
      # Some vars
      k3s_state: uninstalled
      k3s_control_node: true

Works for me.

https://asciinema.org/a/OkABYUy8ZHlRbFaz1iGZmOCBg

For 3rd node, did you metioned k3s_control_node: true when k3s_state: installed?
the link you provided, it seems different. If you enable control node, this should appear in k get nodes . But in your output, this is not there.

[Slowdive-Aideron]

Hey Xan,
Thank you for your help. I'm actually working with David.
In our scenario the actual hostname is different from what we have defined in the inventory.
So inventory_hostname resolves to kube-0 but the output of the command kubectl get nodes gives the actual hostnames -npi--stage-app-02.example.de

So in the when condition - item in kubectl_get_nodes_result.stdout we are comparing apples and oranges :) Will post solution if i find one.

[Slowdive-Aideron]

One fix is to just not use kube-0,kube-1 etc and just put the hostname there