Spurious Deferred node password secret validation failed: secrets "NODENAME" already exists when restarting servers

[brandond]

Environmental Info:
K3s Version:
n/a

Node(s) CPU architecture, OS, and Version:
n/a

Cluster Configuration:
n/a

Describe the bug:

When restarting a server node, deferred node password secret validation may attempt to get the node password secret from an unsynchronized secret cache. When the secret is not found, it attempts to create it, but then gets an "already exists" error because it does exist, it's just not cached yet.

I1029 16:34:55.939401 1734 event.go:307] "Event occurred" object="node-az-3" fieldPath="" kind="Node" apiVersion="" type="Warning" reason="NodePasswordValidationFailed" message="Deferred node password secret validation failed: secrets \"node-az-3.node-password.rke2\" already exists"

Steps To Reproduce:
Restart K3s on a node with many secrets.

Expected behavior:
Secret is found and validated

Actual behavior:
Server log and event indicating node password secret already exists

Additional context / logs:
We should confirm that we aren't caching too many secrets here just to ensure that the node password secrets are available. If we still want to use a cache here, we should validate that it is synchronized before trying to read from it.

k3s/pkg/nodepassword/nodepassword.go

Line 56 in b83f803

secret, err := secretClient.Cache().Get(metav1.NamespaceSystem, name)

• Ref: Deferred node password secret validation failed and complete. rancher/rke2#7309 (comment)