Struggling to get talos cluster configured. #1373

cannfoddr · 2024-03-15T14:48:59Z

cannfoddr
Mar 15, 2024

Hi, I love the look of this project and am working through the setup setps. I have the repo cloned to my own template have setup all the keys ets, installed docker for the devcontainer.

I have edited the config file (to the best of my ability) and can run task configure and have pushed to my repo.

I am planning to use talos on proxmox and have created 3 talos VMs. I think my original attempts were using predicatable interface naming which I think mkaes things complicated. I went to the talos build factory and configured an ISO with the net.ifnames=0 kernel arg and I now get VMs with eth0 at every boot.

When I run talos bootstrap I can see some activity in the talos VM but I eventually end up with messages saying unable to communicate cert errors or some such. I can see my DNS settings have been added from the config.

Clearly I am not sure how to troubleshoot this further. I have build a standalone 6 node cluster manually on my Proxmox cluster using hand crafted config files which is nicely bootstrapped and talking to kubectl. Not done anything with it yet.

How do I go about troubleshooting the tamplate?

onedr0p · 2024-03-15T15:21:00Z

onedr0p
Mar 15, 2024
Maintainer

Hey it looks like you got pretty far at least :)

I am planning to use talos on proxmox and have created 3 talos VMs. I think my original attempts were using predicatable interface naming which I think mkaes things complicated. I went to the talos build factory and configured an ISO with the net.ifnames=0 kernel arg and I now get VMs with eth0 at every boot.

This is pretty specific to VMs and makes sense, on bare metal you have more options to select the interface you want without having to set net.ifnames=0 https://www.talos.dev/v1.6/reference/configuration/v1alpha1/config/#Config.machine.network

If you could share more information (template config file, talhelper config, etc..) it would be helpful in-order to debug.

0 replies

cannfoddr · 2024-03-15T16:00:23Z

cannfoddr
Mar 15, 2024
Author

Some files....

talconfig.yaml

# yaml-language-server: $schema=https://raw.githubusercontent.com/budimanjojo/talhelper/master/pkg/config/schemas/talconfig.json
---
# renovate: datasource=docker depName=ghcr.io/siderolabs/installer
talosVersion: v1.6.5
# renovate: datasource=docker depName=ghcr.io/siderolabs/kubelet
kubernetesVersion: v1.29.2

clusterName: &cluster home-kubernetes
endpoint: https://192.168.1.35:6443
clusterPodNets:
  - "10.69.0.0/16"
clusterSvcNets:
  - "10.96.0.0/16"
additionalApiServerCertSans: &sans
  - "192.168.1.35"
  - 127.0.0.1 # KubePrism
additionalMachineCertSans: *sans
cniConfig:
  name: none

nodes:
  - hostname: "talos-001"
    ipAddress: "192.168.1.115"
    installDisk: "/dev/sda"
    talosImageURL: factory.talos.dev/installer/
    controlPlane: true
    networkInterfaces:
      - interface: eth0
        dhcp: false
        addresses:
          - "192.168.1.115/24"
        mtu: 1500
        routes:
          - network: 0.0.0.0/0
            gateway: "192.168.1.1"
        vip:
          ip: "192.168.1.35"
  - hostname: "talos-002"
    ipAddress: "192.168.1.114"
    installDisk: "/dev/sda"
    talosImageURL: factory.talos.dev/installer/
    controlPlane: true
    networkInterfaces:
      - interface: eth0
        dhcp: false
        addresses:
          - "192.168.1.114/24"
        mtu: 1500
        routes:
          - network: 0.0.0.0/0
            gateway: "192.168.1.1"
        vip:
          ip: "192.168.1.35"
  - hostname: "talos-003"
    ipAddress: "192.168.1.116"
    installDisk: "/dev/sda"
    talosImageURL: factory.talos.dev/installer/
    controlPlane: true
    networkInterfaces:
      - interface: eth0
        dhcp: false
        addresses:
          - "192.168.1.116/24"
        mtu: 1500
        routes:
          - network: 0.0.0.0/0
            gateway: "192.168.1.1"
        vip:
          ip: "192.168.1.35"

patches:
  # Configure containerd
  - |-
    machine:
      files:
        - op: create
          path: /etc/cri/conf.d/20-customization.part
          content: |-
            [plugins."io.containerd.grpc.v1.cri"]
              enable_unprivileged_ports = true
              enable_unprivileged_icmp = true
            [plugins."io.containerd.grpc.v1.cri".containerd]
              discard_unpacked_layers = false
            [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
              discard_unpacked_layers = false

  # Disable search domain everywhere
  - |-
    machine:
      network:
        disableSearchDomain: true

  # Enable cluster discovery
  - |-
    cluster:
      discovery:
        registries:
          kubernetes:
            disabled: false
          service:
            disabled: false

  # Configure kubelet
  - |-
    machine:
      kubelet:
        extraArgs:
          image-gc-low-threshold: 50
          image-gc-high-threshold: 55
          rotate-server-certificates: true
        nodeIP:
          validSubnets:
            - "192.168.1.0/24"

  # Force nameserver
  - |-
    machine:
      network:
        nameservers:
          - 192.168.1.10
          - 192.168.1.11

  # Configure NTP
  - |-
    machine:
      time:
        disabled: false
        servers:
          - time.cloudflare.com

  # Custom sysctl settings
  - |-
    machine:
      sysctls:
        fs.inotify.max_queued_events: 65536
        fs.inotify.max_user_watches: 524288
        fs.inotify.max_user_instances: 8192

  # Mount openebs-hostpath in kubelet
  - |-
    machine:
      kubelet:
        extraMounts:
          - destination: /var/openebs/local
            type: bind
            source: /var/openebs/local
            options:
              - bind
              - rshared
              - rw

  # Disable predictable NIC naming
  - |-
    machine:
      install:
        extraKernelArgs:
          - net.ifnames=0


controlPlane:
  patches:
    # Cluster configuration
    - |-
      cluster:
        allowSchedulingOnMasters: true
        controllerManager:
          extraArgs:
            bind-address: 0.0.0.0
        proxy:
          disabled: true
        scheduler:
          extraArgs:
            bind-address: 0.0.0.0

    # ETCD configuration
    - |-
      cluster:
        etcd:
          extraArgs:
            listen-metrics-urls: http://0.0.0.0:2381
          advertisedSubnets:
            - "192.168.1.0/24"

    # Disable default API server admission plugins.
    - |-
      - op: remove
        path: /cluster/apiServer/admissionControl

    # Enable K8s Talos API Access
    - |-
      machine:
        features:
          kubernetesTalosAPIAccess:
            enabled: true
            allowedRoles:
              - os:admin
            allowedKubernetesNamespaces:
              - system-upgrade

Not sure what else might be useful?

2 replies

onedr0p Mar 15, 2024
Maintainer

factory.talos.dev/installer/

one issue I see ... you are missing the schematic id on talosImageURL, you need to generate and set a schematic id (see the config.yaml)

cannfoddr Mar 15, 2024
Author

Brilliant that has got the cluster bootstrapped

cannfoddr · 2024-03-15T17:18:22Z

cannfoddr
Mar 15, 2024
Author

I'm just trying to figure out:

flux-system home-kubernetes https://github.com/cannfoddr/proxmox-talos-cluster.git 7s False failed to configure authentication options: failed to get secret 'flux-system/github-deploy-key': secrets "github-deploy-key" not found

I added the key originally to GitHub with the wrong name. I've readied but I still can't seem to get past this

The secret doesn't seem to exist

⬢ [Docker] ❯ kubectl get secrets -A
NAMESPACE     NAME                                         TYPE                            DATA   AGE
flux-system   cluster-secrets                              Opaque                          1      51s
flux-system   sops-age                                     Opaque                          1      52s
kube-system   bootstrap-token-45lgzk                       bootstrap.kubernetes.io/token   4      7m4s
kube-system   sh.helm.release.v1.cilium.v1                 helm.sh/release.v1              1      6m27s
kube-system   sh.helm.release.v1.kubelet-csr-approver.v1   helm.sh/release.v1              1      6m28s

2 replies

onedr0p Mar 15, 2024
Maintainer

https://github.com/onedr0p/cluster-template/tree/main?tab=readme-ov-file#-stage-6-install-flux-in-your-cluster

Did you apply it? see the red pin

cannfoddr Mar 16, 2024
Author

I am up and running - I changed all my VMs to have fixed IP assignments on my DMZ network which seems to have streamlined the setup. I have enabled CF and switched to prod certs. I now need to figure out exactly what I have got and how it differs from my homegrown techno Tim based K3s setup with fluxcd. Loose plan is to switch that over to the tacos cluster running in the DMZ.

I need to figure out the DNS piece - I am using technetium DNS server as an internal DNS and need to figure out how to setup a split for the internal vip

Thanks for all the help

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Struggling to get talos cluster configured. #1373

{{title}}

Replies: 3 comments 4 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Struggling to get talos cluster configured. #1373

cannfoddr Mar 15, 2024

Replies: 3 comments · 4 replies

onedr0p Mar 15, 2024 Maintainer

cannfoddr Mar 15, 2024 Author

onedr0p Mar 15, 2024 Maintainer

cannfoddr Mar 15, 2024 Author

cannfoddr Mar 15, 2024 Author

onedr0p Mar 15, 2024 Maintainer

cannfoddr Mar 16, 2024 Author

cannfoddr
Mar 15, 2024

Replies: 3 comments 4 replies

onedr0p
Mar 15, 2024
Maintainer

cannfoddr
Mar 15, 2024
Author

onedr0p Mar 15, 2024
Maintainer

cannfoddr Mar 15, 2024
Author

cannfoddr
Mar 15, 2024
Author

onedr0p Mar 15, 2024
Maintainer

cannfoddr Mar 16, 2024
Author