Qubes fwupd upstream

[macpijan]

@tlaurion

How is heads currently versioned? In the QubesOS script https://github.com/3mdeb/qubes-fwupd/blob/master/src/qubes_fwupd_heads.py#L91 we check for the current version and want to download an update from LVFS only if the version is more recent. We can see that the latest tag is from 2017 though. We have used the LOCALVERSION field to inert the version information for testing purposes.

The Python script drops the update file to the boot partition. Then, the flash-gui.sh script was extended to look for updates there in the first place.

The update process is described here: https://github.com/3mdeb/qubes-fwupd/blob/master/doc/heads_udpate.md

[tlaurion]

@macpijan Heads is versioned per commit ID (as of now) which is inserted at build time under /etc/config
https://github.com/osresearch/heads/blob/master/Makefile#L105
https://github.com/osresearch/heads/blob/master/Makefile#L596

Let me know if we should change something

[tlaurion]

@macpijan I see that the script searches for a ROM inside a known path.
Maybe we should implement a hook inside of /boot integrity validation?

Normal boot codepath entry point for /boot integrity validation

verify_global_hashes() local fonction

check_config imported global fonction

So that if a new firmware is dropped in path, verify_global_hashes could see that the past ROM that was user signed changed, and then check ROM signature against heads imported distinguished distro key (oem_distro_keys instead)?

[Asiderr]

Heads is versioned per commit ID (as of now) which is inserted at build time under /etc/config

@tlaurion Currently we are parsing the Heads version from the fwupdmgr hwids:

https://github.com/fwupd/fwupd/blob/7fc7da3999a3bb0d7e75b4e72993783d82836329/src/fu-tool.c#L1424

Currently, the qubes-fwupd is looking for the line:

BiosVersion: CBET4000 <version> heads

The version number is set in the LOCALVERSION variable. If there is no version tag, it assumes that bios is older than the one that exists in the lvfs.

Let me know if we should change something.

In our case, the easiest to handle would be some release cycle with the version numbers. Every release should be uploaded to the lvfs.

[tlaurion]

@Asiderr can you pinpoint to experts, documentation so that we can move this forward?

My understanding here is that we have to change heads rolling release to a commited time based release, where CircleCI would need to push artifacts back into github, where measured matching (reproducible hashes for different hosts/OS builds) builds would be pushed.

I have no idea right now how to push that change forward, and do not have the rights over Heads repo to do such changes where @osresearch didn't replied to messages sent over slack on the matter after the opening of that PR.

Intuition says that changing manually the version under coreboot config for each board is not the way to go.
I'm clueless on what steps needs to be taken here so that

• this first step is done (versioning)

• then how those versioned artifacts would then be pushed back to github releases for that version, automatically.

Opened an issue a long time ago on that matter. Will edit when found again.

[tlaurion]

@Asiderr I also understand that (actions needed, in short)

• some fwupd code needs to be upstreamed to fwupd @hughsie

• some other qubesos related code needs to be upstreamed to QubesOS @marmarek

• versioning scheme of heads needs to change and CIs configs changed to uploads versions are pushed back to github to be picked up and pushed to lvfs (@Asiderr + @osresearch + @macpijan discussion?)

[Asiderr]

@tlaurion

can you pinpoint to experts, documentation so that we can move this forward?

The latest documentation about the heads update can be found here:
https://github.com/3mdeb/qubes-fwupd/blob/master/doc/heads_udpate.md

Here is the class that handles the heads update process:
https://github.com/3mdeb/qubes-fwupd/blob/master/src/qubes_fwupd_heads.py

Intuition says that changing manually the version under coreboot config
for each board is not the way to go.

You're right, it should be automated somehow.

some fwupd code needs to be upstreamed to fwupd
some other qubesos related code needs to be upstreamed to QubesOS

The source code of the wrapper should be placed in the fwupd/fwupd repo.
Packing scripts should be upstreamed to the QubesOS.

versioning scheme of heads needs to change and CIs configs changed to uploads
versions are pushed back to github to be picked up and pushed to lvfs

That's right.

[tlaurion]

@Asiderr

The source code of the wrapper should be placed in the fwupd/fwupd repo.
Packing scripts should be upstreamed to the QubesOS.

You do the PRs?

[Asiderr]

@tlaurion We're discussing the way of the upstream now:
https://groups.google.com/g/fwupd/c/u5JEIQO_rp8
Once the PRs are created, I'll link them.

[tlaurion]

linked to #839 #859

[tlaurion]

@Asiderr important question on FWUPD #859 (comment)

[tlaurion]

@Asiderr

So following merge of https://github.com/osresearch/heads/pull/859/files#diff-18936189b28399cf48703d0c1ec1df33e57c559de2a12f4438be00e6813bdb68R40-R41

We now have, injected in coreboot config automatically for each built board:

• CONFIG_LOCALVERSION
• CONFIG_MAINBOARD_SMBIOS_PRODUCT_NAME

And the filenames for final produced rom images can be seen, and downloaded for testing of FWUPD usage of CONFIG_MAINBOARD_SMBIOS_PRODUCT_NAME here in ~2 hours from now (since heads modules changes, triggering a new CicrlceCI build only on musl-cross-make CircleCi cache)

Any comment?
Next steps will happen under #571 to:

• Determine next steps to create releases
• Build artifacts and be able to compare hashes.txt for same board from different hosts (docker images)
• Define a process to confirm reproducibility from there
• Define how LVFS images will get uploaded from there
• Any other consideration i'm missing here.

[Asiderr]

@tlaurion Naming and versioning look fine and I'll include the changes in the wrapper.

Define how LVFS images will get uploaded from there

For testing purposes, I'll put manually the cabinet archives on the LVFS. But it's temporary and we have to ask @hughsie if there is any possibility to automate the upload process (I don't see any information in the documentation on this topic).

[hughsie]

is any possibility to automate the upload process

We have robot users that are able to upload automatically; Dell uses this to automate firmware being uploaded to the LVFS as it's built in their pipeline. https://gitlab.com/fwupd/lvfs-website/-/blob/master/contrib/example.py has an example upload if that helps.

[tlaurion]

@macpijan Any news?

[macpijan]

@tlaurion Sorry for the delay. Our plan is to go back to it next week and finish by the end of the year.

[tlaurion]

@macpijan any updates and links would be greatly appreciated!

[macpijan]

@Asiderr Please provide links

[Asiderr]

@tlaurion @macpijan The qubes-wrapper has been merged recently in the fwupd tree:
fwupd/fwupd#2710
I can see that #789 is solved, thus we can swap to the Qubes R4.1.
Now, we should focus on the release process, and we should bring the Heads binaries to the LVFS.

[tlaurion]

@Asiderr @macpijan There is QubesOS/qubes-issues#6792 blocking while a test rom seem to resolve all sandy/ivy bridge issues.

I now have a testing platform to test QubesOS 4.1 deployment on.

Contacted @osresearch to have current Heads repo on a release, with binaries coming from CircleCI uploaded to that github release following https://circleci.com/blog/publishing-to-github-releases-via-circleci/

[tlaurion]

Ping @osresearch

[Asiderr]

@tlaurion I updated flash-gui.sh and now it supports current Heads versioning (#859). Also, I added necessary changes to the fwupd (fwupd/fwupd#4745).

I created a Qubes OS CI in fwupd that generates binary packages, and we are currently discussing the possibility of upstreaming to YUM and APT (fwupd/fwupd#4744).

The last thing we need to think about is the release process of the Heads. The Heads binaries must be uploaded to the LFVS.

[hughsie]

The Heads binaries must be uploaded to the LFVS

Can you use the API? https://lvfs.readthedocs.io/en/latest/upload.html#automatic-uploads

[tlaurion]

@Asiderr: some concerns about content of updates dir and lack of verification of that content prior of proposing end-user for firmware upgrades

[tlaurion]

Question from what will be dropped under /boot/updates//

• Do we have a hashes.txt coming from the builds?
• Is that hashes.txt detached signed with OEM key and provided as well?
• Should we think of adding a OEM directory to add public keys under OEM directory?
  • Should we add validation prior of proposing the ROM to the user and only propose if valid?
  • The same logic then booting from ISO could easily be borrowed here, where ISO is validated against a seperate gpg --home= instance relative to known distro keys

Thoughts?

[tlaurion]

Otherwise, any rom being dropped under /boot/updates would be considered a valid firmware update.
@MrChromebox

[MrChromebox]

Otherwise, any rom being dropped under /boot/updates would be considered a valid firmware update.
@MrChromebox

that's really no different from what we do now from USB

[tlaurion]

@MrChromebox well, right now upgrading from USB considered intentional. Now anything pushing updates to that directory would leave user thinking that firmware is authentic and valid and would prompt the user to flash without further validation from Heads.