Every Kairos version other than alpine fails on Failed to mount sysroot.mount - sysroot.

[firstdorsal]

Hi, I am trying to netboot kairos with the described method here

This worked fine before but not anymore.

I created this script to try out different versions:

#!/bin/bash

set -euo pipefail

#KAIROS_VERSION="v3.0.0-alpha1"
#K3S_VERSION="k3sv1.29.0+k3s1"
#OS="ubuntu-23.10"
# overlayfs: failed to resolve /run/overlayfs': -2
# Failed to mount sysroot.mount - sysroot.

#KAIROS_VERSION="v2.4.3"
#K3S_VERSION="k3sv1.28.2+k3s1"
#OS="ubuntu-23.04"
# overlayfs: failed to resolve /run/overlayfs': -2
# Failed to mount sysroot.mount - sysroot.

#KAIROS_VERSION="v3.0.0-alpha1"
#K3S_VERSION="k3sv1.29.0+k3s1"
#OS="opensuse-tumbleweed"
# secondary node can't join primary

KAIROS_VERSION="v2.4.3"
K3S_VERSION="k3sv1.28.2+k3s1"
OS="alpine-3.18"


BASENAME="kairos-${OS}-standard-amd64-generic-${KAIROS_VERSION}-${K3S_VERSION}"


BASEURL="https://github.com/kairos-io/kairos/releases/download/${KAIROS_VERSION}/${BASENAME}"


wget -nc "${BASEURL}-kernel"
wget -nc "${BASEURL}-initrd"
wget -nc "${BASEURL}.squashfs"

printf "Using config: ${1}\n"

# This will start the pixiecore server.
# Any machine that depends on DHCP to netboot will be send the specified files and the cmd boot line.
docker run \
  --rm --name pixiecore --net=host -v $PWD:/files quay.io/pixiecore/pixiecore \
    boot /files/${BASENAME}-kernel /files/${BASENAME}-initrd --cmdline="rd.neednet=1 ip=dhcp rd.cos.disable root=live:{{ ID \"/files/${BASENAME}.squashfs\" }} netboot nodepair.enable config_url={{ ID \"/files/config-${1}.yaml\" }} console=tty1 console=ttyS0 console=tty0"

with the two files:

config-primary.yaml

#cloud-config

# Define the user accounts on the node.
users:
    - name: "kairos" # The username for the user.
      passwd: "kairos" # The password for the user.
      ssh_authorized_keys: # A list of SSH keys to add to the user's authorized keys.
          - github:firstdorsal # A key from the user's GitHub account.

# Enable K3s on the node.
k3s:
    enabled: true # Set to true to enable K3s.
    args:
        - --cluster-init
        - --tls-san=192.168.122.99
    env:
        K3S_TOKEN: "&rHeF4YWTSHVZS7RA9auT@v^wX4ezf3wEtFQ^CDnSw9qD24cv7L@RHn@_Z`AfdFNv3rxkaa@NYKVaXTbp4gMg$Qk~%R_rQ@bepva"

hostname: node-1

install:
    # Device for automated installs
    device: "/dev/vda"
    # Reboot after installation
    reboot: true
    auto: true
    #image: "docker:docker.io/firstdorsal/kairos-test"
    #encrypted_partitions:
    #    - COS_PERSISTENT
    #partitions:
    #    persistent:
    #        size: 5000 # 5GB this is the encrypted partition where etcd data is stored, should be bigger in prod
    #        fs: ext4
    #extra-partitions:
    #    - name: extra_userdata
    #      size: 10 # use the rest of the disk, this is for longhorn to claim, longhorn will handle the encryption
    #      # this currently fails https://github.com/kairos-io/kairos/issues/2159
    #      fs: ext4

kcrypt:
    challenger:
        # External KMS Server address. This must be reachable by the node
        challenger_server: "http://192.168.122.216:30000"
        # (optional) Custom Non-Volatile index to use to store encoded blobs
        nv_index: ""
        # (optional) Custom Index for the RSA Key pair
        c_index: ""
        # (optional) Custom TPM device
        tpm_device: ""

config-secondary.yaml

#cloud-config
users:
    - name: "kairos" # The username for the user.
      passwd: "kairos" # The password for the user.
      ssh_authorized_keys: # A list of SSH keys to add to the user's authorized keys.
          - github:firstdorsal # A key from the user's GitHub account.

hostname: node-{{ trunc 4 .MachineID }}

install:
    # Device for automated installs
    device: "/dev/vda"
    # Reboot after installation
    reboot: true
    auto: true
    #image: "docker:docker.io/firstdorsal/kairos-test"

k3s:
    enabled: true # Set to true to enable K3s.
    args:
        - --server https://node-1:6443
    env:
        K3S_TOKEN: "&rHeF4YWTSHVZS7RA9auT@v^wX4ezf3wEtFQ^CDnSw9qD24cv7L@RHn@_Z`AfdFNv3rxkaa@NYKVaXTbp4gMg$Qk~%R_rQ@bepva"

excuted with: bash install.sh primary and bash install.sh secondary respectively

It boots fine until it logs:

overlayfs: failed to resolve /run/overlayfs': -2
Failed to mount sysroot.mount - sysroot.

for every system running systemd

opensuse-tumbleweed continues the installation process but then fails on starting k3s on the second node:

it works somewhat when executing sudo /usr/bin/k3s server --server https://node-1:6443 on the second node, and it receives data from the first but never shows up when running kubectl get nodes

The environment is qemu/kvm.

I tried newer versions, older versions added and removed the virtual TPM from the VMs etc. nothing seems to work :/

This is what pixieboot shows, also it requests the files again on every boot even after the install has finished.

I also don't understand why it requests other-0 and other-1 but never the squashfs

[mauromorales]

Haven't looked into it in detail but could it be the rd.live.overlay.overlayfs stanza in the cmdline?

[firstdorsal]

Yes that fixed the problem. Thank you!
Maybe it should be included in the documentation?

I used it like this:

#!/bin/bash

set -euo pipefail

KAIROS_VERSION="v2.5.0"
K3S_VERSION="k3sv1.29.0+k3s1"
OS="debian-bookworm"

BASENAME="kairos-${OS}-standard-amd64-generic-${KAIROS_VERSION}-${K3S_VERSION}"

BASEURL="https://github.com/kairos-io/kairos/releases/download/${KAIROS_VERSION}/${BASENAME}"

wget -nc "${BASEURL}-kernel"
wget -nc "${BASEURL}-initrd"
wget -nc "${BASEURL}.squashfs"

printf "Using config: ${1}\n"

# This will start the pixiecore server.
# Any machine that depends on DHCP to netboot will be send the specified files and the cmd boot line.
docker run \
  --rm --name pixiecore --net=host -v $PWD:/files quay.io/pixiecore/pixiecore \
    boot /files/${BASENAME}-kernel /files/${BASENAME}-initrd --cmdline="rd.neednet=1 rd.live.overlay.overlayfs=1 ip=dhcp rd.cos.disable root=live:{{ ID \"/files/${BASENAME}.squashfs\" }} netboot nodepair.enable config_url={{ ID \"/files/config-${1}.yaml\" }} console=tty1 console=ttyS0 console=tty0"