Unable to locate matching firmware on Incus 6.5

[adamcstephens]

Required information

• Distribution: NixOS
• Distribution version: Unstable 20240906.574d1ea
• The output of "incus info" or if that fails:
  • Kernel version: 6.6.49
  • LXC version: 6.0.1
  • Incus version: 6.5
  • Storage backend in use: zfs

Issue description

Unable to start two VMs after upgrade from 6.4 to 6.5. Others VMs and containers work, so I suspect this is a problem with these two specific VMs.

❯ incus start link1
Error: Unable to locate matching firmware: [{Code:/nix/store/yy6drc448c7s61a16jcychp9x28ix2zn-incus-ovmf/OVMF_CODE.4MB.fd Vars:/nix/store/yy6drc448c7s61a16jcychp9x28ix2zn-incus-ovmf/OVMF_VARS.4MB.fd} {Code:/nix/store/yy6drc448c7s61a16jcychp9x28ix2zn-incus-ovmf/OVMF_CODE.fd Vars:/nix/store/yy6drc448c7s61a16jcychp9x28ix2zn-incus-ovmf/OVMF_VARS.fd} {Code:/nix/store/yy6drc448c7s61a16jcychp9x28ix2zn-incus-ovmf/OVMF_CODE.fd Vars:/nix/store/yy6drc448c7s61a16jcychp9x28ix2zn-incus-ovmf/OVMF_VARS.fd} {Code:/nix/store/yy6drc448c7s61a16jcychp9x28ix2zn-incus-ovmf/OVMF_CODE.fd Vars:/nix/store/yy6drc448c7s61a16jcychp9x28ix2zn-incus-ovmf/OVMF_VARS.fd}]

We set the EDK2 path explicitly using INCUS_EDK2_PATH=/nix/store/yy6drc448c7s61a16jcychp9x28ix2zn-incus-ovmf to a directory. Note that these are themselves symlinked elsewhere.

❯ ls -1 /nix/store/yy6drc448c7s61a16jcychp9x28ix2zn-incus-ovmf
OVMF_CODE.4MB.fd -> /nix/store/615kgb34yjj7cl20bgv5anx1gb9m8c2s-OVMF-202408-fd/FV/OVMF_CODE.fd
OVMF_CODE.fd -> /nix/store/p78yh96dinh6s7xggysy24p53k88cdci-OVMF-202408-fd/FV/OVMF_CODE.fd
OVMF_VARS.4MB.fd -> /nix/store/615kgb34yjj7cl20bgv5anx1gb9m8c2s-OVMF-202408-fd/FV/OVMF_VARS.fd
OVMF_VARS.4MB.ms.fd -> /nix/store/615kgb34yjj7cl20bgv5anx1gb9m8c2s-OVMF-202408-fd/FV/OVMF_VARS.fd
OVMF_VARS.fd -> /nix/store/p78yh96dinh6s7xggysy24p53k88cdci-OVMF-202408-fd/FV/OVMF_VARS.fd
OVMF_VARS.ms.fd -> /nix/store/p78yh96dinh6s7xggysy24p53k88cdci-OVMF-202408-fd/FV/OVMF_VARS.fd

❯ ls -1 /nix/store/615kgb34yjj7cl20bgv5anx1gb9m8c2s-OVMF-202408-fd/FV/OVMF_CODE.fd /nix/store/615kgb34yjj7cl20bgv5anx1gb9m8c2s-OVMF-202408-fd/FV/OVMF_VARS.fd
/nix/store/615kgb34yjj7cl20bgv5anx1gb9m8c2s-OVMF-202408-fd/FV/OVMF_CODE.fd
/nix/store/615kgb34yjj7cl20bgv5anx1gb9m8c2s-OVMF-202408-fd/FV/OVMF_VARS.fd

Steps to reproduce

I don't have clear steps to reproduce. These two VMs have existed for a long time, but they are not able to start after upgrading to 6.5.

Information to attach

• Any relevant kernel output (dmesg)
• Container log (incus info NAME --show-log)
• Container configuration (incus config show NAME --expanded)
• Main daemon log (at /var/log/incus/incusd.log)
• Output of the client with --debug
• Output of the daemon with --debug (alternatively output of incus monitor --pretty while reproducing the issue)

On a whim, I tried moving the qemu.nvram out of the way for one VM, and the VARS file was copied in and symlinked allowing the VM to start. I'm not sure such a workaround is desirable, but if it is then the error message should provide better guidance.

[root@blank:/var/lib/incus/virtual-machines]# ll guide1/
total 25
-rw-r--r-- 1 root   root    696 Oct 24  2022 agent-client.crt
-rw------- 1 root   root    288 Oct 24  2022 agent-client.key
-rw-r--r-- 1 root   root    733 Oct 24  2022 agent.crt
-rw------- 1 root   root    288 Oct 24  2022 agent.key
-r-------- 1 root   root   2470 Sep  8 08:31 backup.yaml
dr-x------ 6 nobody root     14 Sep  8 08:31 config
-r--r--r-- 1 root   root    168 Dec 31  1969 metadata.yaml
drwxr-xr-x 3 root   root      3 Dec 31  1969 nix
-r--r--r-- 1 root   root      0 Dec 31  1969 nix-path-registration
-rw------- 1 nobody root 540672 Sep  8 08:31 OVMF_VARS.4MB.fd
lrwxrwxrwx 1 root   root     16 Sep  8 08:31 qemu.nvram -> OVMF_VARS.4MB.fd
-rw------- 1 nobody root 131072 Sep  7 01:19 qemu.nvram.bak

But this one is still broken:

[root@blank:/var/lib/incus/virtual-machines]# ll link1/
total 15
-rw-r--r-- 1 root   root    700 Nov 22  2022 agent-client.crt
-rw------- 1 root   root    288 Nov 22  2022 agent-client.key
-rw-r--r-- 1 root   root    733 Nov 22  2022 agent.crt
-rw------- 1 root   root    288 Nov 22  2022 agent.key
-r-------- 1 root   root   2872 Sep  7 10:20 backup.yaml
dr-x------ 6 nobody root     14 Sep  8 08:32 config
-r--r--r-- 1 root   root    168 Dec 31  1969 metadata.yaml
drwxr-xr-x 3 root   root      3 Dec 31  1969 nix
-r--r--r-- 1 root   root      0 Dec 31  1969 nix-path-registration
-rw------- 1 nobody root 131072 Sep  7 01:27 qemu.nvram

[stgraber]

qemu.nvram is definitely intended to always be a symlink to a file whose name should match one of those available on the system, so we can locate an appropriate firmware.

I think this broken environment was caused by a bug which we fixed with some firmware rework in 6.5...

In the case where qemu.nvram is a plain file, we actually have no idea what firmware file we need to use for it. Your best bet at that point is to reset things.

The more user friendly way to reset the firmware code+nvram for a VM is to do:

• incus config set NAME security.secureboot=false
• incus config unset NAME security.secureboot

That's because enabling/disabling secureboot requires a full reset of the NVRAM, which should then clear things and set them back up properly.

[stgraber]

The ownership of qemu.nvram by nobody:nogroup is also a bit puzzling, really not sure what got it to be created in that way.

[adamcstephens]

The more user friendly way to reset the firmware code+nvram for a VM is to do:

* `incus config set NAME security.secureboot=false`

* `incus config unset NAME security.secureboot`

Unfortunately this didn't seem to work in this scenario. It copied the OVMF VARS but didn't fix the file symlink.

❯ sudo ls -l /var/lib/incus/virtual-machines/link1/
total 22
<snip>
-rw-r--r-- 1 root   root 540672 Sep  8 14:37 OVMF_VARS.4MB.fd
-rw------- 1 nobody root 131072 Sep  7 01:27 qemu.nvram

I removed this qemu.nvram and started it again and incus recreated the symlink successfully. It looks like across all my hosts only these two VMs had the issue. They were likely migrated from LXD many months ago.

I'm happy to close if you think there's no action here.

[stgraber]

Interesting, we may need to be slightly more aggressive on qemu.nvram then to make sure to switch it to a symlink if it's not already one.

[banafshehft]

same issue here. i get this error:

Error: Unable to locate matching firmware: [{Code:/usr/share/ovmf/x64/OVMF_CODE.4m.fd Vars:/usr/share/ovmf/x64/OVMF_VARS.4m.fd} {Code:/usr/share/ovmf/x64/OVMF_CODE.fd Vars:/usr/share/ovmf/x64/OVMF_VARS.fd} {Code:/usr/share/ovmf/x64/OVMF_CODE.4m.fd Vars:/usr/share/ovmf/x64/OVMF_VARS.4m.fd} {Code:/usr/share/ovmf/x64/OVMF_CODE.fd Vars:/usr/share/ovmf/x64/OVMF_VARS.fd} {Code:/usr/share/ovmf/x64/OVMF_CODE.4m.fd Vars:/usr/share/ovmf/x64/OVMF_VARS.4m.fd} {Code:/usr/share/ovmf/x64/OVMF_CODE.fd Vars:/usr/share/ovmf/x64/OVMF_VARS.fd}]

al so i have : -rw------- 1 incus root 128K Sep 11 17:46 qemu.nvram
i am using arch-linux and update incus 6.5 on friday

[lzslab]

incus in nixos
update incus to 6.6

incus start test
Error: Unable to locate matching firmware: []

journalctl -u incus
level=error msg="Unable to run feature checks during QEMU [initi](initialization: Unable to locate a UEFI firmware)

[adamcstephens]

That looks unrelated to this issue. Please open an issue in nixpkgs first before reporting upstream.

[lukasz-zaroda]

I had the same issue on Incus 6.0.2 which is supposed to be stable. I think it's a critical issue. Thankfully, setting/unsetting secureboot helped. In my case, I just run apt upgrade on Ubuntu 22.04 and rebooted the machine. My vms didn't come up.