Confused on proper key protection/how to check if initramfs is properly secured

[maanloper]

To start offf:
My ubuntu server system works great, ZBM boots fine (using rEFInd, what a gem is that!), unlocking pool either direct or over SSH works great, ZBM cloning/rollback works great, so first of all a big thank you to who made all of this possible!

However, I'm stuck on checking if I properly secured my keyfile. Hope someone can help me out - please add code snippets I can run to check, I'm just learning about initramfs/kernels/etc.

I installed ZFS on Root with ZBM by following the main guide guides/ubuntu/noble-uefi with these options:

• Encrypted root
• Build ZBM from source (with remote access using dracut-crypt-ssh)
• rEFInd

The questions I have:

1. In the guide it shows:

echo 'SomeKeyphrase' > /etc/zfs/zroot.key
chmod 000 /etc/zfs/zroot.key

This is still in the live environment; wouldn't you need to set it while chrooted?

2. There is a note in the guide, which says:

Because the encryption key is stored in /etc/zfs directory, it will automatically be copied into the system initramfs.

However, in native-encryption this is shown:

On systems that use dracut, the key for zroot can be added to initramfs images by running:

echo 'install_items+=" /etc/zfs/zroot.key "' > /etc/dracut.conf.d/zfs-keys.conf

Wouldn't that put the key in the ZBM? As warned in another info-block slightly above it:

Never place encryption keys inside a custom ZFSBootMenu image! The ZFSBootMenu image will typically be installed on an unencrypted partition with minimal or no access restrictions. If an encryption key is placed in such a location, anybody with access to the system will be able to read your passphrase.

Also, isn't the initramfs on an unencrypted boot partition, readable to everyone? To me it feels like the ubuntu-guide and the native-encryption pages contradict each other a bit. Granted, I don't really understand initramfs, so maybe I'm missing something obvious here.

3. My most important question: I want to check if my key is not being leaked. From native-encryption:

When adding encryption keys to initramfs images, always ensure that the resulting images are not readable by any user other than root. Recent versions of dracut and mkinitcpio ensure this by default with umask of 0077. Users with read access to your initramfs image will be able to read your ZFS key file even if it has mode 000 in the image; always confirm for your self that the initramfs is protected!

How do I check this? This is my /boot:

   89 -rw-r--r-- 1 root root   287413 Nov 30 18:21 config-6.8.0-51-generic
    4 drwxr-xr-x 3 root root     4096 Jan  1  1970 efi
    1 lrwxrwxrwx 1 root root       27 Jan  8 15:39 initrd.img -> initrd.img-6.8.0-51-generic
75137 -rw-r--r-- 1 root root 76990546 Jan 10 22:57 initrd.img-6.8.0-51-generic
    1 lrwxrwxrwx 1 root root       27 Jan  8 15:39 initrd.img.old -> initrd.img-6.8.0-51-generic
 1869 -rw------- 1 root root  9072978 Nov 30 18:21 System.map-6.8.0-51-generic
    1 lrwxrwxrwx 1 root root       24 Jan  8 15:39 vmlinuz -> vmlinuz-6.8.0-51-generic
14465 -rw------- 1 root root 14969224 Nov 30 19:09 vmlinuz-6.8.0-51-generic
    1 lrwxrwxrwx 1 root root       24 Jan  8 15:39 vmlinuz.old -> vmlinuz-6.8.0-51-generic

And /boot/efi/EFI/ZBM:

95376 -rwxr-xr-x 1 root root 97664422 Jan 10 18:42 initramfs-2.3.0_10.img
95376 -rwxr-xr-x 1 root root 97664935 Jan 10 09:03 initramfs-2.3.0_8.img
95376 -rwxr-xr-x 1 root root 97662188 Jan 10 09:34 initramfs-2.3.0_9.img
    4 -rwxr-xr-x 1 root root       94 Jan 10 18:42 refind_linux.conf
14620 -rwxr-xr-x 1 root root 14969224 Jan 10 18:42 vmlinuz-2.3.0_10
14620 -rwxr-xr-x 1 root root 14969224 Jan 10 09:03 vmlinuz-2.3.0_8
14620 -rwxr-xr-x 1 root root 14969224 Jan 10 09:34 vmlinuz-2.3.0_9

On advice of the almighty ChatGPT I also ran lsinitrd /boot/initrd.img-6.8.0-51-generic:

Image: /boot/initrd.img-6.8.0-51-generic: 74M
========================================================================
Early CPIO image
========================================================================
drwxr-xr-x   2 root     root            0 Oct 19  2023 .
drwxr-xr-x   2 root     root            0 Oct 19  2023 kernel
drwxr-xr-x   2 root     root            0 Oct 19  2023 kernel/x86
drwxr-xr-x   2 root     root            0 Oct 19  2023 kernel/x86/microcode
-rw-r--r--   1 root     root        76166 Oct 19  2023 kernel/x86/microcode/AuthenticAMD.bin
========================================================================
Version:

Arguments:
dracut modules:
========================================================================
drwxr-xr-x   2 root     root            0 Aug 29 12:00 kernel
drwxr-xr-x   2 root     root            0 Aug 29 12:00 kernel/x86
drwxr-xr-x   2 root     root            0 Aug 29 12:00 kernel/x86/microcode
drwxr-xr-x   2 root     root            0 Aug 29 12:00 kernel/x86/microcode/.enuineIntel.align.0123456789abc
-rw-r--r--   1 root     root      7979008 Aug 29 12:00 kernel/x86/microcode/GenuineIntel.bin

And also lsinitrd /boot/efi/EFI/ZBM/initramfs-2.3.0_10.img:

Image: /boot/efi/EFI/ZBM/initramfs-2.3.0_10.img: 94M
========================================================================
Early CPIO image
========================================================================
drwxr-xr-x   3 root     root            0 Jan 10 18:42 .
-rw-r--r--   1 root     root            2 Jan 10 18:42 early_cpio
drwxr-xr-x   3 root     root            0 Jan 10 18:42 kernel
drwxr-xr-x   3 root     root            0 Jan 10 18:42 kernel/x86
drwxr-xr-x   2 root     root            0 Jan 10 18:42 kernel/x86/microcode
-rw-r--r--   1 root     root        76166 Jan 10 18:42 kernel/x86/microcode/AuthenticAMD.bin
-rw-r--r--   1 root     root     14073856 Jan 10 18:42 kernel/x86/microcode/GenuineIntel.bin
========================================================================
Version: dracut-060+5-1ubuntu3.2

Arguments:  -f --confdir '/etc/zfsbootmenu/dracut.conf.d' -q

dracut modules:
bash
modsign
console-setup
i18n
network-legacy
network
crypt-ssh
bcache
crypt
dm
kernel-modules
kernel-modules-extra
kernel-network-modules
lvm
mdraid
multipath
nvdimm
overlay-root
qemu
qemu-net
zfsbootmenu
iscsi
lunmask
nbd
nfs
rootfs-block
terminfo
udev-rules
virtiofs
usrmount
base
fs-lib
shutdown
========================================================================
<whole bunch of files>

"lsinitrd /boot/efi/EFI/ZBM/initramfs-2.3.0_10.img | grep zfs" gives:
Arguments:  -f --confdir '/etc/zfsbootmenu/dracut.conf.d' -q
zfsbootmenu
-rw-r--r--   1 root     root          243 Jan 10 18:41 etc/zfsbootmenu.conf
-rwxr-xr-x   1 root     root         2285 Jan  8 15:40 libexec/zfsbootmenu-diff
-rwxr-xr-x   1 root     root         2522 Jan  8 15:40 libexec/zfsbootmenu-help
-rwxr-xr-x   1 root     root         7821 Jan  8 15:40 libexec/zfsbootmenu-init
-rwxr-xr-x   1 root     root          128 Jan  8 15:40 libexec/zfsbootmenu-input
-rwxr-xr-x   1 root     root         1377 Jan  8 15:40 libexec/zfsbootmenu-preview
-rwxr-xr-x   1 root     root          719 Jan  8 15:40 libexec/zfsbootmenu-run-hooks
-rwxr-xr-x   1 root     root         7119 Jan  8 15:40 usr/bin/zfsbootmenu
-rwxr-xr-x   1 root     root         3291 Jan  8 15:40 usr/bin/zfs-chroot
lrwxrwxrwx   1 root     root           11 Jan 10 18:41 usr/bin/zfs -> ../sbin/zfs
-rwxr-xr-x   1 root     root         7414 Jan  8 15:40 usr/lib/dracut/hooks/cmdline/95-zfsbootmenu-parse-commandline.sh
-rwxr-xr-x   1 root     root           39 Jan  8 15:40 usr/lib/dracut/hooks/initqueue/finished/99-zfsbootmenu-ready-chk.sh
-rwxr-xr-x   1 root     root           57 Jan  8 15:40 usr/lib/dracut/hooks/initqueue/settled/99-zfsbootmenu-ready-set.sh
-rwxr-xr-x   1 root     root         1402 Jan  8 15:40 usr/lib/dracut/hooks/pre-mount/90-zfsbootmenu-preinit.sh
drwxr-xr-x   2 root     root            0 Jan 10 18:41 usr/lib/modules/6.8.0-51-generic/kernel/zfs
-rw-r--r--   1 root     root        58052 Nov 30 18:21 usr/lib/modules/6.8.0-51-generic/kernel/zfs/spl.ko.zst
-rw-r--r--   1 root     root      1869627 Nov 30 18:21 usr/lib/modules/6.8.0-51-generic/kernel/zfs/zfs.ko.zst
-rw-r--r--   1 root     root          324 Sep  5 19:38 usr/lib/udev/rules.d/90-zfs.rules
-rw-r--r--   1 root     root        55520 Sep  5 19:38 usr/lib/x86_64-linux-gnu/libzfs_core.so.3.0.0
lrwxrwxrwx   1 root     root           20 Jan 10 18:42 usr/lib/x86_64-linux-gnu/libzfs_core.so.3 -> libzfs_core.so.3.0.0
-rw-r--r--   1 root     root       495536 Sep  5 19:38 usr/lib/x86_64-linux-gnu/libzfs.so.4.1.0
lrwxrwxrwx   1 root     root           15 Jan 10 18:42 usr/lib/x86_64-linux-gnu/libzfs.so.4 -> libzfs.so.4.1.0
-rw-r--r--   1 root     root         2594 Jan  8 15:40 usr/lib/zfsbootmenu-completions.sh
-rwxr-xr-x   1 root     root        51717 Jan  8 15:40 usr/lib/zfsbootmenu-core.sh
-rwxr-xr-x   1 root     root         6019 Jan  8 15:40 usr/lib/zfsbootmenu-kcl.sh
-rwxr-xr-x   1 root     root        18500 Jan  8 15:40 usr/lib/zfsbootmenu-ui.sh
-rwxr-xr-x   1 root     root          752 Sep  5 19:38 usr/sbin/fsck.zfs
-rwxr-xr-x   1 root     root        22840 Sep  5 19:38 usr/sbin/mount.zfs
-rwxr-xr-x   1 root     root       147632 Sep  5 19:38 usr/sbin/zfs
-rw-r--r--   1 root     root        21677 Jan  8 15:40 usr/share/docs/help-files/132/zfsbootmenu.7.ansi
-rw-r--r--   1 root     root        23509 Jan  8 15:40 usr/share/docs/help-files/52/zfsbootmenu.7.ansi
-rw-r--r--   1 root     root        22165 Jan  8 15:40 usr/share/docs/help-files/92/zfsbootmenu.7.ansi
drwxr-xr-x   3 root     root            0 Jan 10 18:41 usr/share/zfs
drwxr-xr-x   3 root     root            0 Jan 10 18:41 usr/share/zfsbootmenu
drwxr-xr-x   2 root     root            0 Jan 10 18:41 usr/share/zfsbootmenu/fonts
-rw-r--r--   1 root     root         7933 Jan  8 15:40 usr/share/zfsbootmenu/fonts/ter-v12n.psf
-rw-r--r--   1 root     root         9450 Jan  8 15:40 usr/share/zfsbootmenu/fonts/ter-v14b.psf
-rw-r--r--   1 root     root        22269 Jan  8 15:40 usr/share/zfsbootmenu/fonts/ter-v20b.psf
-rw-r--r--   1 root     root        26365 Jan  8 15:40 usr/share/zfsbootmenu/fonts/ter-v24b.psf
-rw-r--r--   1 root     root        30461 Jan  8 15:40 usr/share/zfsbootmenu/fonts/ter-v28b.psf
-rw-r--r--   1 root     root        34557 Jan  8 15:40 usr/share/zfsbootmenu/fonts/ter-v32b.psf
drwxr-xr-x   2 root     root            0 Jan  8 15:39 usr/share/zfs/compatibility.d
lrwxrwxrwx   1 root     root           11 Sep  5 19:38 usr/share/zfs/compatibility.d/2018 -> compat-2018
lrwxrwxrwx   1 root     root           11 Sep  5 19:38 usr/share/zfs/compatibility.d/2019 -> compat-2019
lrwxrwxrwx   1 root     root           11 Sep  5 19:38 usr/share/zfs/compatibility.d/2020 -> compat-2020
lrwxrwxrwx   1 root     root           11 Sep  5 19:38 usr/share/zfs/compatibility.d/2021 -> compat-2021
-rw-r--r--   1 root     root          211 Sep  5 19:38 usr/share/zfs/compatibility.d/compat-2018
-rw-r--r--   1 root     root          246 Sep  5 19:38 usr/share/zfs/compatibility.d/compat-2019
-rw-r--r--   1 root     root          246 Sep  5 19:38 usr/share/zfs/compatibility.d/compat-2020
-rw-r--r--   1 root     root          306 Sep  5 19:38 usr/share/zfs/compatibility.d/compat-2021
-rw-r--r--   1 root     root          227 Sep  5 19:38 usr/share/zfs/compatibility.d/freebsd-11.0
lrwxrwxrwx   1 root     root           12 Sep  5 19:38 usr/share/zfs/compatibility.d/freebsd-11.1 -> freebsd-11.0
-rw-r--r--   1 root     root          275 Sep  5 19:38 usr/share/zfs/compatibility.d/freebsd-11.2
-rw-r--r--   1 root     root          287 Sep  5 19:38 usr/share/zfs/compatibility.d/freebsd-11.3
lrwxrwxrwx   1 root     root           12 Sep  5 19:38 usr/share/zfs/compatibility.d/freebsd-11.4 -> freebsd-11.3
lrwxrwxrwx   1 root     root           12 Sep  5 19:38 usr/share/zfs/compatibility.d/freebsd-12.0 -> freebsd-11.3
lrwxrwxrwx   1 root     root           12 Sep  5 19:38 usr/share/zfs/compatibility.d/freebsd-12.1 -> freebsd-11.3
lrwxrwxrwx   1 root     root           12 Sep  5 19:38 usr/share/zfs/compatibility.d/freebsd-12.2 -> freebsd-11.3
lrwxrwxrwx   1 root     root           12 Sep  5 19:38 usr/share/zfs/compatibility.d/freebsd-12.3 -> freebsd-11.3
lrwxrwxrwx   1 root     root           12 Sep  5 19:38 usr/share/zfs/compatibility.d/freebsd-12.4 -> freebsd-11.3
lrwxrwxrwx   1 root     root           19 Sep  5 19:38 usr/share/zfs/compatibility.d/freebsd-13.0 -> openzfs-2.1-freebsd
lrwxrwxrwx   1 root     root           19 Sep  5 19:38 usr/share/zfs/compatibility.d/freebsd-13.1 -> openzfs-2.1-freebsd
lrwxrwxrwx   1 root     root           19 Sep  5 19:38 usr/share/zfs/compatibility.d/freebsd-13.2 -> openzfs-2.1-freebsd
lrwxrwxrwx   1 root     root           12 Sep  5 19:38 usr/share/zfs/compatibility.d/freenas-11.0 -> freebsd-11.0
lrwxrwxrwx   1 root     root           12 Sep  5 19:38 usr/share/zfs/compatibility.d/freenas-11.1 -> freenas-11.0
lrwxrwxrwx   1 root     root           12 Sep  5 19:38 usr/share/zfs/compatibility.d/freenas-11.2 -> freebsd-11.2
lrwxrwxrwx   1 root     root           12 Sep  5 19:38 usr/share/zfs/compatibility.d/freenas-11.3 -> freebsd-11.3
-rw-r--r--   1 root     root          216 Sep  5 19:38 usr/share/zfs/compatibility.d/freenas-9.10.2
-rw-r--r--   1 root     root          353 Sep  5 19:38 usr/share/zfs/compatibility.d/grub2
-rw-r--r--   1 root     root          505 Sep  5 19:38 usr/share/zfs/compatibility.d/openzfs-2.0-freebsd
-rw-r--r--   1 root     root          509 Sep  5 19:38 usr/share/zfs/compatibility.d/openzfs-2.0-linux
-rw-r--r--   1 root     root          511 Sep  5 19:38 usr/share/zfs/compatibility.d/openzfs-2.1-freebsd
-rw-r--r--   1 root     root          515 Sep  5 19:38 usr/share/zfs/compatibility.d/openzfs-2.1-linux
-rw-r--r--   1 root     root          584 Sep  5 19:38 usr/share/zfs/compatibility.d/openzfs-2.2
lrwxrwxrwx   1 root     root           11 Sep  5 19:38 usr/share/zfs/compatibility.d/openzfs-2.2-freebsd -> openzfs-2.2
lrwxrwxrwx   1 root     root           11 Sep  5 19:38 usr/share/zfs/compatibility.d/openzfs-2.2-linux -> openzfs-2.2
-rw-r--r--   1 root     root          239 Sep  5 19:38 usr/share/zfs/compatibility.d/openzfsonosx-1.7.0
-rw-r--r--   1 root     root          299 Sep  5 19:38 usr/share/zfs/compatibility.d/openzfsonosx-1.8.1
-rw-r--r--   1 root     root          401 Sep  5 19:38 usr/share/zfs/compatibility.d/openzfsonosx-1.9.3
lrwxrwxrwx   1 root     root           18 Sep  5 19:38 usr/share/zfs/compatibility.d/openzfsonosx-1.9.4 -> openzfsonosx-1.9.3
lrwxrwxrwx   1 root     root           19 Sep  5 19:38 usr/share/zfs/compatibility.d/truenas-12.0 -> openzfs-2.0-freebsd
lrwxrwxrwx   1 root     root            7 Sep  5 19:38 usr/share/zfs/compatibility.d/ubuntu-18.04 -> zol-0.7
lrwxrwxrwx   1 root     root            7 Sep  5 19:38 usr/share/zfs/compatibility.d/ubuntu-20.04 -> zol-0.8
lrwxrwxrwx   1 root     root           17 Sep  5 19:38 usr/share/zfs/compatibility.d/ubuntu-22.04 -> openzfs-2.1-linux
-rw-r--r--   1 root     root           81 Sep  5 19:38 usr/share/zfs/compatibility.d/zol-0.6.1
-rw-r--r--   1 root     root          166 Sep  5 19:38 usr/share/zfs/compatibility.d/zol-0.6.4
-rw-r--r--   1 root     root          197 Sep  5 19:38 usr/share/zfs/compatibility.d/zol-0.6.5
-rw-r--r--   1 root     root          267 Sep  5 19:38 usr/share/zfs/compatibility.d/zol-0.7
-rw-r--r--   1 root     root          398 Sep  5 19:38 usr/share/zfs/compatibility.d/zol-0.8

Hope anyone can shed some light on this for the less advanced users. Thanks!

[zdykstra]

The key has to be created in the live environment because you're also creating the pool with your encryption root in the live environment. When you get further along in the install process, one of the steps is to copy /etc/zfs into the chroot. The file will not be part of ZFSBootMenu initramfs or EFI file if you're using Dracut, because ZFSBootMenu doesn't use the system-wide dracut configuration directory, nor does it use the Dracut ZFS module.

The directions work as written.

[maanloper]

Thank you for the quick reply! The ZBM-part is clear to me now.
Could you maybe help me out in how I can check this:

When adding encryption keys to initramfs images, always ensure that the resulting images are not readable by any user other than root. Recent versions of dracut and mkinitcpio ensure this by default with umask of 0077. Users with read access to your initramfs image will be able to read your ZFS key file even if it has mode 000 in the image; always confirm for your self that the initramfs is protected!

I want to be absolutely sure I did not make a mistake somewhere. Because when doing update-initramfs -c -k all it states just above it:

Because the encryption key is stored in /etc/zfs directory, it will automatically be copied into the system initramfs.

So that means the key is copied into the initramfs (which resides on unencrypted /boot)? Just making sure I'm not putting the strongest lock on my front door ever, while leaving the back door wide open without noticing :)

[zdykstra]

/boot is a directory under /. It's encrypted if your root dataset is encrypted.

[maanloper]

I assumed because it said /boot/efi that /boot was also mounted at sda1:

root@servername:# lsblk
NAME     MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda        8:0    0 111.8G  0 disk
├─sda1     8:1    0   512M  0 part  /boot/efi
├─sda2     8:2    0     4G  0 part
│ └─swap 252:0    0     4G  0 crypt [SWAP]
└─sda3     8:3    0 107.3G  0 part
sdb        8:16   0 465.8G  0 disk
├─sdb1     8:17   0 465.8G  0 part
└─sdb9     8:25   0     8M  0 part
sr0       11:0    1  1024M  0 rom

I think I just got confused, since df /boot and df /boot/efi show you are 100% correct:

root@servername:# df /boot
Filesystem              1K-blocks    Used Available Use% Mounted on
zroot/ROOT/ubuntu_24.04 107050112 1601664 105448448   2% /

root@servername:# df /boot/efi
Filesystem     1K-blocks   Used Available Use% Mounted on
/dev/sda1         523244 351316    171928  68% /boot/efi

My apologies for the confusion and thank you for taking the time to help me understand.
Really liking ZBM + ZFS so far - truly lets me learn/try out things and within a second I can revert any mistakes!

Edit: you gave me the "DUH"-moment of the week :D

[zdykstra]

Glad everything is working for you! Consider looking into a tool that takes automatic snapshots for you - both of your OS and your home directory.