What are differences between v1.28.14 (candidate) and v1.28.13 (stable)? Patch Release Notes for each Revision would be great!

[Aaron-Ritter]

Summary

None of the v1.28.13 come online after system got rebooted, only when i tried to upgrade to the candidate v1.28.14 release it came back online.

prod-m1 Ready 35m v1.28.14
prod-m2 Ready 258d v1.28.14
prod-m3 NotReady 258d v1.28.13
prod-m4 NotReady 33m v1.28.13

Could you please provide me what are differences between v1.28.14 (candidate) and v1.28.13 (stable) are?

And as a sub-sequential request could you please generally create and publish patch release notes?

What Should Happen Instead?

They should become Ready after a restart.

I should be able to know what each patch release contains in changes and be able to look them up online.

Reproduction Steps

None of the nodes came online anymore after a system restart, non of the following tests worked it was in a constant restart loop similar to the modprobe nf_conntrack issue:

• modprobe hack
• manual restarts
• disabling apparmor
• leaving the cluster and adding the node back again
• leaving the cluster removing the snap and reinstalling it
• etc.

I checked if the last microk8s update and compare to last system boot. There was a monthly snap update in between so i suspected the new snap microk8s v1.28.13 version caused the problems and checked if an alternative was available and refreshed snap to v1.28.14 (candidate).

Can you suggest a fix?

v1.28.14 seems fixing it.

publish patch release notes.

Are you interested in contributing with a fix?

yes for testing

[louiseschmidtgen]

Hi @Aaron-Ritter,

thank you for raising your issue.

For the difference between v1.28.13 and v1.28.14 I can only point you to the Kubernetes upstream changelog.

I suspect that you are on an older version of v1.28.13 prior to the back-port fix of this issue.
The associated PR for 1.28 (classic) MicroK8s is: #4651.
This is the new tag in k8s-dqlite v1.1.11 with the patch: canonical/k8s-dqlite#161.

We hope to communicate changes better in the future!

[Aaron-Ritter]

Hi @louiseschmidtgen,

is my assumption correct that if i use a snap v1.28.13 it is not the full picture because it still could be that there is an update available for it with the exact same version tag but the release date and (Revision) is different?.

So your snap releases are for each vN.N.N YYYY-MM-DD (NNNN) change where:

• vN.N.N follows the upstream version https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.28.md#changelog-since-v12813
• But couple own changes and patches in to it, not only once but possible multiple times which causes the YYYY-MM-DD (NNNN) to change.

It would be really great to have patch notes similar to the upstream ones but for each individual snap release (Revision).

Because i can see the backport to canonical/k8s-dqlite#161 but then again i cant see the upgrade to v1.28.14: https://github.com/canonical/microk8s/commits/1.28/. And i wouldn't know, by side assuming based on the date stamp, if what ever got backported is actually in the snap release.

So in my case it would have been possibly enough to just run a refresh pull the latest v1.28.13 release with a new Revision.

[louiseschmidtgen]

Hello @Aaron-Ritter,

you are correct: refreshing to the latest revision of v1.28.13 would've likely resolved your issue, assuming that your snap revision is pinned. To learn more about managing snap updates visit this page.

We have automation in place to update all k8s tracks (here v1.28) with the latest Kubernetes patch: https://github.com/canonical/microk8s/blob/1.28/build-scripts/components/kubernetes/version.sh and publish those builds. This is the reason why you are not seeing a PR bumping the k8s patch for v1.28.

For maintenance purposes we backport security and bug fixes to all maintained tracks (here v1.28), resulting in a new revision available in the stable channel of v1.28. Unless your snap is pinned to a revision you will receive these updates.

We will discuss your request for revision notes your suggestions are much appreciated!

[Aaron-Ritter]

Thanks @louiseschmidtgen, this clarifies it further, we pinned it to channel=1.28/stable and limited the autoupdate to a specific weekday once a month.

My takeaway for now is to get the snap info microk8s and check the revision of the installed version vs the available one, to see if there are any potential changes to microk8s. And checking the version differences via Kubernetes upstream changelog for any kubernetes changes.