Unable to successfully put LUKS key into TPM #603

vanceb · 2019-08-18T14:02:16Z

Having a problem mentioned in #566 . I have managed to install Heads and Qubes. I can boot into Qubes using the "Disk Recovery Password". The problem comes when I try to set a default boot option and store the disk recovery key in the TPM.

Using the menu to set the default appears to go well. On reboot, and selecting "Default boot" all goes well until I get asked for the "unlock password". Once entered I get the following:

Error PCR mismatch from TPM_Unseal
PCR-00: <20 hex bytes>
PCR-01: <20 hex bytes>
...
PCR-07 <20 hex bytes>
Unable to unseal disk encryption key

I understand from this that the PCRs have changed since they were "Sealed" into the TPM. I have tried to reset the PCR values by doing:

seal-totp

<gets a qrcode>

unseal-totp

Error PCR mismatch from TPM_Unseal
unable to unseal totp secret

Also when booting I notice that I get notification of PCR changes, specifically PCR 7 and PCR 5. I am not sure if this is related to the "PCR mismatch"???

I think I am missing something, but not sure where to go from here...

The text was updated successfully, but these errors were encountered:

vanceb · 2019-08-19T14:23:45Z

I have tried to reset this a number of times without success. I have tried to use the Heads boot menus and follow the prompts, and the process seems fine, but whenn I come to reboot and use the default boot option it fails:

As I can't copy/paste from the Heads machine this will be a summary:

*** Found verified kexec boot params
gpg: Signature made <date>
gpg:                     Using RSA Key <key sig>
gpg: Good signature from <email address>
*** Found verified kexec boot params
*** Scanning for unsigned boot options
*** Checking verified boot hash file
*** Verified boot hashes
65323284: <hex digits>
/tmp/counter-65323284: OK
*** Checking verified default boot hash file
*** Verified default boot hashes
*** Executing default boot for Qubes, with Xen hypervisor:
New value of PCR[6]: <hex digits>
Enter unlock password (blank to abort)
Error PCR mismatch from TPM_Unseal
PCR-00: <20 hex bytes>
PCR-01: <20 hex bytes>
...
PCR-04: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
...
PCR-07 <20 hex bytes>
Unable to unseal disk encryption key

The thing that worries me in the above is that PCR[6] has changed just before I enter my unlock password. From reading http://osresearch.net/Keys.html PCR[6] is the LUKS header. I can see no reason why this should change at all during boot or any other time, unless I change the passwords? Secondly, why is PCR[4] All zeros?

Any suggestions?

tlaurion · 2019-08-19T20:26:49Z

PCR[4] is extended when getting to the recovery console to invalidate measurements.

My past experience with disk unlock key not being unsealed correctly was caused by usb devices drivers being loaded though insmod which calls insmod script(not busybox insmod directly), which modifies measurements.

Sealing and unsealing requires to have same measurements, and loading usb modules will interfere with those measurements.

If going straight to boot options and setting a new boot default, it should work as designed.

tlaurion · 2019-08-19T20:28:29Z

PCR[6] is modified by measuring the LUKS header.

vanceb · 2019-08-20T14:21:13Z

Why does the LUKS header change just before I enter the unlock password? I think it is this value changing that causes a failure to unseal.

As far as I am aware I am not causing the loading of additional USB modules. I did think it could have been the Yubikey being inserted to sign the updated /boot after selecting a new default, but I don't notice a PCR change when I insert theYubikey. After the failure to unseal the TPM I drop myself into the recovery shell to list the USB modules:

/boot # lsmod
Module         Size       Used by     Not tainted
xhci_pci        16384    -
xhci_hcd       98304    -
ehci_pci        16384    -
ehci_hcd       45056    -

Plugging in the Yubikey and running lsmod again shows no difference.

vanceb · 2019-08-20T14:24:45Z

FYI I am using the Master branch not the release:
commit 20d79f5

I have tried "going straight to boot options and setting new boot default" a couple of times. Here is a transcript of my last attempt:

Clean boot from power down
Presented with menu screen showing correct totp value
Choose third option Settings-->
From settings menu choose first option Other Boot Options
From other boot options menu choose Show OS boot menu
- Drops to terminal view before showing Select your boot option menu with multiple boot options
Chooses first boot option Qubes,_with_Xen_hypervisor_[/xen-4.8.5-7.fc25.gz]
Choose second option Make Qubes, with Xen hypervisor the default
- Drops into terminal

Saving a default will modify the disk.  Proceed? (y/n):  y
Do you want to reseal a disk key into the TPM [yN]: y
    Reading all physical volumes.  This may take a while
Encrypted LVM group? (was ''): <leaves blank>
/dev/sda2: UUID="<uuid>"
/dev/sda1: UUID="<uuid>"
Encrypted devices? (was '/dev/sda2/'): /dev/sda2
Running kexec-save-key with params: -s -p /boot /dev/sda2
Enter disk recovery key: <enters recovery key>
New disk unlock password for booting: <enters password>
Repeat unlock code: <repeats password>
++++++ /dev/sda2: Removing old key slot
++++++ /dev/sda2: Adding key
New value of PCR[6]: <hex digits>
Please confirm that your GPG card is inserted [Y/n]: y
...
< gpg card details listed >
...
Please unlock the card
PIN: <enters PIN>
gpg: Signature made....
...
+++ Found verified kexec boot params
+++ Saved defaults to device
+++ Rebooting to start the new default boot option

Computer reboots
Presented with boot menu with correct totp
Select first option Default boot

Drops to shell

** Found verified kexec boot params
gpg: Signature made <date>
gpg:                     Using RSA Key <key sig>
gpg: Good signature from <email address>
*** Found verified kexec boot params
*** Scanning for unsigned boot options
*** Checking verified boot hash file
*** Verified boot hashes
65323284: <hex digits>
/tmp/counter-65323284: OK
*** Checking verified default boot hash file
*** Verified default boot hashes
*** Executing default boot for Qubes, with Xen hypervisor:
New value of PCR[6]: <hex digits>
Enter unlock password (blank to abort)
Error PCR mismatch from TPM_Unseal
PCR-00: <20 hex bytes>
PCR-01: <20 hex bytes>
...
PCR-04: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
...
PCR-07 <20 hex bytes>
Unable to unseal disk encryption key

Any more ideas?

vanceb · 2019-08-20T14:34:49Z

Hmmm, may not be PCR[6] after all. Looking back at the value of PCR[6] just after the LUKS headers were changed when setting the default, matched the value just before the request for unlock password when trying to default boot. But if it hasn't changed from what the TPM expects why is it notified that it has changed?

Should the TPM be resealed with the new PCRs (specifically PCR[6]) after the default boot option has been set? I answered for the TPM to reseal the disk key, but does this also update the PCRs stored at the same time? I can see no log line stating that this has happened as part of changing the default boot option?

vanceb · 2019-08-21T07:28:34Z

Just thinking through this a bit more...

At stage 2 of above, I am presented with a valid TOTP value. This must mean that the PCRs are OK at this point as the TPM must unseal the TOTP secret.

During stage 7 (Making Qubes the default) it says that PCR[6] has changed. I am guessing that this means that PCR[6] was previously at a default value and has just been populated with a value related to the LUKS headers which have just been read. The value of PCR[6] at this point is the same as the value shown when it fails to unseal the TPM at stage 10.

If my thinking is correct then at stage 2 the value of PCR[6] is at a default value, maybe 0, and the TPM unlocks OK with this value. When I try to boot the default at stage 10 above, the LUKS headers are read and PCR[6] is populated with the value based on the LUKS headers (which is different from the default value which worked at stage 2), and as the PCRs are now different it fails to unseal the disk key.

If I am correct then we somehow need to read the LUKS headers so we populate PCR[6] with the correct value before sealing the disk key into the TPM???

Is this an issue with code in the master branch, or am I barking up the wrong tree?

tlaurion · 2019-08-21T08:24:19Z

I'll see code differences with my QubesOS certified branch. Haven't tested master for a little while now and they have diverted.

Tonux599 · 2019-08-24T01:40:38Z

Had a lot of problems with this issue as well. After messing around a bit the only thing that worked for me was specifying CONFIG_BOOTSCRIPT=/bin/generic-init in the board config and following the instructions on the wiki.

Is it possible something in the gui init could be causing this issue?

vanceb · 2019-08-24T07:37:19Z

As a work-around I can set the default boot if I choose not to put the LUKS key into the TPM. I am then prompted for the LUKS key durring OS boot instead of the TPM disk key in Heads boot. I suspect I am then missing a check of the LUKS headers by the TPM before boot. So I would be vulnerable to LUKS header modification.

I may try CONFIG_BOOTSCRIPT=/bin/generic_init when I get a chance.

tlaurion · 2019-09-03T16:51:10Z

I'll look into this next week. @kylerankin: has some changes changed the behavior, not taking into consideration the disk unlock boot path?

…

On August 24, 2019 1:40:39 AM UTC, Tonux599 ***@***.***> wrote: Had a lot of problems with this issue as well. After messing around a bit the only thing that worked for me was specifying CONFIG_BOOTSCRIPT=/bin/generic-init and following the instructions on the wiki. Is it possible something in the gui init could be causing this issue? -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #603 (comment)

-- Sent from /e/ Mail

kylerankin · 2019-09-03T19:58:30Z

@tlaurion Given I don't use that feature I'm not really sure, however it's certainly possible that PCR measurements get updated and changed after the initial boot verification and retrieving a LUKS secret later on when the user tells Heads to boot.

I can only imagine that the UI will introduce more changes in the future instead of less, as we all continue to add ease-of-use features that launch scripts behind the scenes, so perhaps the LUKS secret should be cached somewhere safely in /tmp beforehand?

tlaurion · 2019-10-08T01:39:20Z

@vanceb can you retry resealing TPM and HOTP through menus instead of the console?

Going in console modifies PCR 4 which won't match normal boot.

Are you using gui-init in board config?

vanceb · 2019-10-08T20:56:30Z

I upgraded to the latest Master Build 5d91d6a

Resealing TOTP works fine through the GUI menu. The problem I was having was getting the LUKS key into the TPM. I have reverted back to using Ubuntu rather than Qubes, and I don't think it will allow me to pass the disk encryption key through from the TPM (I am sure I read it somewhere). But I have tried to set a new default and include the disk key by using the menu. This fails before asking me for the disk key, without really providing an error message and puts me back at the boot menu. If I don't try to include the disk key it asks me for the GPG key and sets the default as it should.

I am not sure whether this is because I am now using Ubuntu?

EDmitry · 2019-12-29T02:24:39Z

I am having the same issue (I think). Here is what I do:

Load recovery shell, run kexec-unseal-key. It fails due to my previous unsuccessful attempts to seal the key, but it does show me the current PCR registers.
Seal new keys using kexec-save-key. The PCR6 correctly changes before sealing the key.
Run kexec-unseal-key again. It will obviously fail, because PCR4 isn't all zeros, but it does show me the value of all the PCR registers so far, all other registers should be what I'd expect at boot.
Reboot and try to load the default option. Asks me for the key passphrase, PCR6 changes to the correct previous value, but fails with "PCR mismatch" again.

As far as I can see, the state of PCR registers is exactly the same that it was at a sealing stage, just with PCR4 being all zeros, which is correct. Something else is probably going on, I don't think the value of registers is wrong. Is there any way to debug this further or get more information?

PS. When sealing I end kexec-save-key at a point when it asks to insert a GPG key. I assume it doesn't matter, because the sealing has already happened before that and /boot/kexec-devices.txt doesn't actually change if I am continuously working with the same LUKS partition.

EDmitry · 2019-12-29T04:43:25Z

Okay, I have figured it out. kexec-seal-key was inputting zeros as PCR5. I think init-gui makes PCR5 values change, so they are not zeros anymore.

@tlaurion looks like your suspicion was correct, it's due to the differences in gui and common load paths.

petermn2 · 2019-12-29T23:08:11Z

Okay, I have figured it out. kexec-seal-key was inputting zeros as PCR5. I think init-gui makes PCR5 values change, so they are not zeros anymore.

@tlaurion looks like your suspicion was correct, it's due to the differences in gui and common load paths.

I've been dealing with this same issue. I just tried the same process as @EDmitry and get the same results. I'm going to try and reflash with the generic-init option and see if I can find out more.

EDmitry · 2019-12-29T23:53:48Z

@petermn2 you don't have to switch to generic-init to fix it. Just boot in recovery shell, and edit /bin/kexec-seal-key (vi /bin/kexec-seal-key). You need to replace the line -ix 5 $pcr_5 \ with -ix 5 X \. Then you would run kexec-save-key as instructed in wiki immediately after and get your TPM key sealed correctly.

Note that after a reboot all changes you've made to kexec-seal-key will be gone, so you would need to edit the script every time you need to reseal the key, unless you reflash the new version of the script permanently.

petermn2 · 2019-12-30T03:28:41Z

@EDmitry, thank you, now I understand what you are saying, and I made the change and it works as you said. I'm looking at the code for keyexec-seal-key and I see where it is setting PCR5 to 0 and I'm trying to figure out the reasoning for the change:

# Now that we have setup the new keys, measure the PCRs
# We don't care what ends up in PCR 6; we just want
# to get the /tmp/luksDump.txt file.  We use PCR16
# since it should still be zero
cat "$KEY_DEVICES" | cut -d\  -f1 | xargs /bin/qubes-measure-luks \
	|| die "Unable to measure the LUKS headers"
luks_pcr=`tpm calcfuturepcr -ix 16 -if /tmp/luksDump.txt`

# Librem Key loads USB modules which changes PCR5.
# In the event Librem Key is enabled, skip verification of PCR5
if [ -x /bin/libremkey_hotp_verification ]; then
  pcr_5="X"
else
  pcr_5="0000000000000000000000000000000000000000"
fi
# Note that PCR 4 needs to be set with the "normal-boot"
# path value, which we do not have right now since we are
# in a recovery shell.
# used to be -ix 4 f8fa3b6e32e7c6fe04c366e74636e505b28f3b0d \
# now just all zeros in a normal boot
# PCR 5 must be all zero since no kernel modules should have
# been loaded during a normal boot, but might have been
# loaded in the recovery shell.
# Otherwise use the current values of the PCRs, which will be read
# from the TPM as part of the sealing ("X").'
tpm sealfile2 \
	-if "$KEY_FILE" \
	-of "$TPM_SEALED" \
	-pwdd "$key_password" \
	-hk 40000000 \
	-ix 0 X \
	-ix 1 X \
	-ix 2 X \
	-ix 3 X \
	-ix 4 0000000000000000000000000000000000000000 \
	-ix 5 $pcr_5 \
	-ix 6 $luks_pcr \
	-ix 7 X \
|| die "Unable to seal secret"'

I think there is a mistake in this, the comment is right the kernal shouldn't be loaded yet, but it still needs to be the measured value not just zeroed out. Or am I missing something?

EDmitry · 2019-12-30T04:19:53Z

@petermn2 gui-init runs enable-usb which in turns loads kernel usb modules, which changes PCR5 (it's all zeroes before any kernel modules are loaded and every module load extends the PCR value). I suspect common-gui doesn't do that. The comment says that it's needed for external keyboards to work. Perhaps this should be optional, I'd prefer no modules loaded unless absolutely necessary. Especially since heads mostly targets laptops.

tlaurion · 2020-01-01T16:32:31Z

@EDmitry @vanceb :
The logic pointed by checking existence of libremkey_hotp_verification

if [ -x /bin/libremkey_hotp_verification ]; then
  pcr_5="X"
else
  pcr_5="0000000000000000000000000000000000000000"

Is that Librem Key/Nitrokey Pro/Nitrokey Storage requires USB modules to be loaded to be able to do HOTP remote attestation at each boot, just after firmware measurements have been taken to remote attest with the USB security dongle.

So here, if we have libremkey_hotp_verification, this means that the board configuration supplied Librem Key config option.

My hypothesis here is that if some USB functions are called (loading modules, modifying PCR5) and in the same run, a LUKS sealing is attempted, upon reboot, without having the same PCR5 measurements because usb modules are not loaded, the LUKS unseal won't match because measurements are not the same. Can you reproduce without calling any USB functions, sealing and unsealing?

EDmitry · 2020-01-01T20:15:11Z

@tlaurion I’ve mentioned before that I’ve found an insmod called from init-gui — https://github.com/osresearch/heads/blob/8af849cadc684259865123195b86317ecfa53528/initrd/bin/gui-init#L132

I’ll see if I can remove this, reflash and verify that PCR5 is all zeroes in that case.

tlaurion · 2020-01-01T22:44:53Z

@MrChromebox : this change caused a regression for non Librem Key users, since the LUKS key sealing still expects PCR5 to be 0

Most Heads users don't expect to have USB modules loaded but when needed ( and don't use USB keyboards. Even less when using Qubes?)

Consequently, enable-usb and usb-init should be called only where needed.

MrChromebox · 2020-01-01T23:17:36Z

@MrChromebox : this change caused a regression for non Librem Key users, since the LUKS key sealing still expects PCR5 to be 0

Most Heads users don't expect to have USB modules loaded but when needed ( and don't use USB keyboards. Even less when using Qubes?)

Consequently, enable-usb and usb-init should be called only where needed.

would guarding 'enable_usb' with 'if (CONFIG_LIBREMKEY)' be sufficient to mitigate?

@MrChromebox Yes and no since it seems that server boards also requires usb support for keyboards. I suppose measurements should be consistent in all case. (#601 (comment))

petermn2 · 2020-01-05T22:09:55Z

@tlaurion @EDmitry @MrChromebox
I've been playing around with this issue this weekend and can confirm @EDmitry's suspicion.
I just ran some tests:

1st - I did a clean build from current (as of today) heads master. I did a fresh install, following wiki directions starting from initial x230-flash externally flashed. From there I used flash.sh to install clean master rom. I sealed the luks key to tpm entered new unlock password and on reboots could not unlock the luks drive with symptoms exactly as described by @EDmitry .

2nd - I did a new clean build from osr/heads master with the following change to /bin/initrd/gui-init:
line 132: #enable_usb
Reflashed, reset TPM, set tpm/totp, set default boot, etc...
Unsealing Qubes luks disk with TPM unlock code works fine every time.

With all the reflashing and resetting from scratch I've been doing I've noticed some additional anomolies/bugs with gui-init that I will post in new issues once I've verified them further.

ensures external USB keyboards are accessible Signed-off-by: Matt DeVillier <[email protected]>

tlaurion · 2020-02-16T22:57:03Z

@MrChromebox : I thought that there was a PR to make the "enable_usb" call dependant of LIBREM_KEY but I can't find it. A lot of users, not depending on the Librem Keys under heads, are facing issues regarding the merge of 4a85c85

MrChromebox · 2020-02-16T23:49:10Z

@tlaurion I don't recall there ever being a determination of that being an acceptable fix, but I'll submit a PR for it tomorrow

tlaurion · 2020-02-17T14:12:20Z

@MrChromebox: USB_KEYBOARD board option for servers seem to be a better fit?

Some (out of tree) servers require use of a USB keyboard, and need the USB kernel modules loaded prior to checking for keypress to enter a recovery console. Since loading the modules affects the value in PRC5 and can cause issues putting a LUKS key in TPM, guard the loading of the USB modules with CONFIG_USB_KEYBOARD and remove the unguarded call from gui-init. This should resolve issues linuxboot#603 and linuxboot#674. Signed-off-by: Matt DeVillier <[email protected]>

tlaurion · 2020-02-19T17:56:18Z

fixed with Fixed with 8e23a54

tlaurion mentioned this issue Jan 26, 2020

Supermicro x11ssh coreboot configuration #601

Draft

tlaurion referenced this issue Feb 16, 2020

gui-init: load USB modules at startup

4a85c85

ensures external USB keyboards are accessible Signed-off-by: Matt DeVillier <[email protected]>

EDmitry mentioned this issue Feb 19, 2020

LUKS KEY SEALING TROUBLESHOOTING LOG AND FIX #674

Closed

MrChromebox mentioned this issue Feb 19, 2020

init: load usb modules for devices using USB keyboard #677

Merged

tlaurion closed this as completed Feb 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to successfully put LUKS key into TPM #603

Unable to successfully put LUKS key into TPM #603

vanceb commented Aug 18, 2019

vanceb commented Aug 19, 2019 •

edited

Loading

tlaurion commented Aug 19, 2019 •

edited

Loading

tlaurion commented Aug 19, 2019

vanceb commented Aug 20, 2019

vanceb commented Aug 20, 2019 •

edited

Loading

vanceb commented Aug 20, 2019 •

edited

Loading

vanceb commented Aug 21, 2019 •

edited

Loading

tlaurion commented Aug 21, 2019

Tonux599 commented Aug 24, 2019 •

edited

Loading

vanceb commented Aug 24, 2019

tlaurion commented Sep 3, 2019 via email

kylerankin commented Sep 3, 2019

tlaurion commented Oct 8, 2019 •

edited

Loading

vanceb commented Oct 8, 2019 •

edited

Loading

EDmitry commented Dec 29, 2019

EDmitry commented Dec 29, 2019

petermn2 commented Dec 29, 2019

EDmitry commented Dec 29, 2019 •

edited

Loading

petermn2 commented Dec 30, 2019 •

edited

Loading

EDmitry commented Dec 30, 2019 •

edited

Loading

tlaurion commented Jan 1, 2020

EDmitry commented Jan 1, 2020

tlaurion commented Jan 1, 2020 •

edited

Loading

MrChromebox commented Jan 1, 2020 •

edited by tlaurion

Loading

petermn2 commented Jan 5, 2020

tlaurion commented Feb 16, 2020

MrChromebox commented Feb 16, 2020

tlaurion commented Feb 17, 2020

tlaurion commented Feb 19, 2020

Unable to successfully put LUKS key into TPM #603

Unable to successfully put LUKS key into TPM #603

Comments

vanceb commented Aug 18, 2019

vanceb commented Aug 19, 2019 • edited Loading

tlaurion commented Aug 19, 2019 • edited Loading

tlaurion commented Aug 19, 2019

vanceb commented Aug 20, 2019

vanceb commented Aug 20, 2019 • edited Loading

vanceb commented Aug 20, 2019 • edited Loading

vanceb commented Aug 21, 2019 • edited Loading

tlaurion commented Aug 21, 2019

Tonux599 commented Aug 24, 2019 • edited Loading

vanceb commented Aug 24, 2019

tlaurion commented Sep 3, 2019 via email

kylerankin commented Sep 3, 2019

tlaurion commented Oct 8, 2019 • edited Loading

vanceb commented Oct 8, 2019 • edited Loading

EDmitry commented Dec 29, 2019

EDmitry commented Dec 29, 2019

petermn2 commented Dec 29, 2019

EDmitry commented Dec 29, 2019 • edited Loading

petermn2 commented Dec 30, 2019 • edited Loading

EDmitry commented Dec 30, 2019 • edited Loading

tlaurion commented Jan 1, 2020

EDmitry commented Jan 1, 2020

tlaurion commented Jan 1, 2020 • edited Loading

MrChromebox commented Jan 1, 2020 • edited by tlaurion Loading

petermn2 commented Jan 5, 2020

tlaurion commented Feb 16, 2020

MrChromebox commented Feb 16, 2020

tlaurion commented Feb 17, 2020

tlaurion commented Feb 19, 2020

vanceb commented Aug 19, 2019 •

edited

Loading

tlaurion commented Aug 19, 2019 •

edited

Loading

vanceb commented Aug 20, 2019 •

edited

Loading

vanceb commented Aug 20, 2019 •

edited

Loading

vanceb commented Aug 21, 2019 •

edited

Loading

Tonux599 commented Aug 24, 2019 •

edited

Loading

tlaurion commented Oct 8, 2019 •

edited

Loading

vanceb commented Oct 8, 2019 •

edited

Loading

EDmitry commented Dec 29, 2019 •

edited

Loading

petermn2 commented Dec 30, 2019 •

edited

Loading

EDmitry commented Dec 30, 2019 •

edited

Loading

tlaurion commented Jan 1, 2020 •

edited

Loading

MrChromebox commented Jan 1, 2020 •

edited by tlaurion

Loading