Talos and k8s version mismatch preventing upgrades

[ianepreston]

I have a three node cluster with three control plane nodes with scheduling allowed. I've been messing around with it for a few months and am finally ready to try upgrading talos and k8s but find myself stuck.

I'm running talos 1.7.6 and kubernetes 1.32.0. I know talos 1.7.6 ships with kubernetes 1.30.3 and to be honest I have no idea how I got myself on 1.32.0. I'm almost certain I haven't run talosctl upgrade-k8s or anything similar although I accept there's good evidence to the contrary.

Accepting that for whatever reason I'm in this situation, I'm unclear if there's an easy way to get out, or if I should wipe the cluster and restart. I don't have anything critical on here, it's just a learning cluster, but I'd like to find out if there's a proper way since the point is to learn things.

I can't upgrade from talos 1.7.6 to 1.7.7 because my kubernetes version is too new:

192.168.40.7: user: warning: [2025-02-25T01:50:22.305577008Z]: Error: pre-flight checks failed: component kubelet version issue: version of Kubernetes 1.32.0 is too new to be used with Talos 1.7.7

I also can't upgrade directly to talos 1.9.4 because my talos version is too old:

192.168.40.7: user: warning: [2025-03-01T14:18:26.856987729Z]: Error: pre-flight checks failed: host version 1.7.6 is too old to upgrade to Talos 1.9.4

I can't rollback kubernetes to 1.30.3, at least from talosctl:

❯ talosctl --nodes 192.168.40.7 upgrade-k8s --to 1.30.3 --dry-run
automatically detected the lowest Kubernetes version 1.32.0
unsupported upgrade path 1.32->1.30 (from "1.32.0" to "1.30.3")

Any suggestions for how I can get myself out of this mess that aren't wiping everything and starting over? I'd also be interested in ideas about how I got into this version mismatch situation to begin with. I installed a base release image of talos on the machines to start. I've added cilium, argocd, cert-manager, external-secrets, traefik, multus, and reloader to the cluster since installing.

[ianepreston]

Trying to find clean ways to at least get out of this I thought I'd see if I can put the machine in maintenance and then run an install or upgrade.

I added an install config patch:

machine:
  install:
    image: factory.talos.dev/installer/f9355a00f82a90f7dda4b0548c181fc96a2bdc14fd5eea61542b90562ed339ae:v1.9.4

put the machine in maintenance and applied the config, but it came back up with my old version of talos and k8s.

I put it back in maintenance and tried to upgrade:

❯ talosctl upgrade --nodes 192.168.40.7 --image factory.talos.dev/installer/f9355a00f82a90f7dda4b0548c181fc96a2bdc14fd5eea61542b90562ed339ae:v1.9.4 --insecure --wait=false
error performing upgrade: rpc error: code = Unimplemented desc = API is not implemented in maintenance mode

Looks like I could do it if I had Omni, but this is just a home cluster. So I think I have to create new boot media and fully reinstall.

Will leave this open a bit to see if there's a smarter way but that's where I'm at for now.