kboot status report for 2023 Q2 #202

bsdimp · 2023-07-07T00:20:07Z

oh, and a little before too, but this is my first one

bsdimp · 2023-07-08T03:50:23Z

Since I'm not selling anything, I don't need r or tm. I don't want to imply I'm selling anything by using them. The other comments look good and I'll likely include them all...

lsalvadore

Thanks for your report. I made a few suggestions too and I have also spotted a few typos.

lsalvadore · 2023-07-09T08:53:25Z

website/content/en/status/report-2023-04-2023-06/kboot.adoc

+
+The old Sony Playstation 3 port used a boot loader called 'kboot' to accomplish this in that environment.
+That code has been greatly expanded, made generic with easily replaceable per-architecture plug ins.
+The normal FreeBSD /boot/loader is built as a Linux binary that reads in the FreeBSD kernel, modules and tunables.


It is probably better to use the filename markup here:

[.filename]#/boot/loader#

Cross-reference https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=266881#c3 re: discovery of markup such as this.

lsalvadore · 2023-07-09T08:54:38Z

website/content/en/status/report-2023-04-2023-06/kboot.adoc

+For UEFI enabled systems, it passes the UEFI memory table, and pointer to UEFI runtime services to the new kernel.
+
+It supports loading files from the host's filesystem, from any loader(8) supported filesystem on the host's block devices (including ZPOOLS that span multiple devices), from ram disk images and from files downloaded over the network.
+Any mix of these is available, so configuraiton overrides can be loaded from the host's filesystem, but the kernel can be loaded from dedicated storage (say NVME) or a ram disk image.


s/configuraiton/configuration/

lsalvadore · 2023-07-09T08:55:48Z

website/content/en/status/report-2023-04-2023-06/kboot.adoc

+It supports loading files from the host's filesystem, from any loader(8) supported filesystem on the host's block devices (including ZPOOLS that span multiple devices), from ram disk images and from files downloaded over the network.
+Any mix of these is available, so configuraiton overrides can be loaded from the host's filesystem, but the kernel can be loaded from dedicated storage (say NVME) or a ram disk image.
+It supports a host console running over stdin/stdout.
+It supports explicit locations like `/dev/nvme0ns1:/boot/loader/gerbil.conf` to be explicit about where to load filesystems from.


[.filename]#/dev/nvme0ns1:/boot/loader/gerbil.conf#

close enough to real filename, but only in the context of the boot loader. And only the new kboot loader.
So I'm unsure what to do with this advice.

lsalvadore · 2023-07-09T08:56:41Z

website/content/en/status/report-2023-04-2023-06/kboot.adoc

+It supports explicit locations like `/dev/nvme0ns1:/boot/loader/gerbil.conf` to be explicit about where to load filesystems from.
+If supports ZFS boot environments, including the boot-once feature.
+
+FreeBSD/aarch64 now can boot from Linux in a LinuxBoot environment, with comparable support and functionality as loader.efi.


[.filename]#loader.efi#

kboot status report for 2023 Q2 #202 (comment)

went with @grahamperrin 's suggestion instead that he linked.

lsalvadore · 2023-07-09T08:59:11Z

website/content/en/status/report-2023-04-2023-06/kboot.adoc

+
+FreBSD/amd64 in progress, maybe 80% done.
+The amd64 boot environment has more quirks and details to get right than the aarch64 environment given its much older lineage.
+The kernel boots part way before panicing because things aren't setup correctly.


I am unsure about "panicing". Maybe it should be "panicking"?

The kernel panics partway through initialization since all the prerequisites from the boot loader have not been discovered and implemented.

lsalvadore · 2023-07-09T08:59:52Z

website/content/en/status/report-2023-04-2023-06/kboot.adoc

+
+Help needed:
+
+ 1. Currently, there is no documentation on how to use loader.kboot, nor on how to create test images.


[.filename]#loader.kboot#

I've updated this with

1. man:loader.kboot[8] needs to be written. It should document how to use loader.kboot, how to create images, and the different use cases that work today.

lsalvadore · 2023-07-09T09:00:21Z

website/content/en/status/report-2023-04-2023-06/kboot.adoc

+ 1. Currently, there is no documentation on how to use loader.kboot, nor on how to create test images.
+ 1. Finish amd64 support.
+ 1. The current elf arch-specific metadata code is copied from efi. +
+Unifying the kboot eand efi copies is needed. +


s/eand/and/

lsalvadore · 2023-07-09T09:00:48Z

website/content/en/status/report-2023-04-2023-06/kboot.adoc

+ 1. Finish amd64 support.
+ 1. The current elf arch-specific metadata code is copied from efi. +
+Unifying the kboot eand efi copies is needed. +
+This is complicated because they aren't identical.


Please do not use contractions:
s/aren't/are not/

reworded as

1. The current elf arch-specific metadata code is copied from efi. + Unifying the kboot and efi copies is needed. + While they are mostly the same, compile-time differences remain that complicate sharing. + In addtion, the build infrastructure makes sharing awkward.

Sponsored by: Netflix

bsdimp · 2023-07-10T06:35:52Z

pushed update that I claim addresses all the feedback here.
Please let me know if I missed anything (I tried to include it all, except trademarks, etc since any such trademark used is done in a descriptive way referring to a family of products, not one specific one, in ways that would be hard to construe as endorsement).
I've also tweaked some overly verbose and passive language
Plus rebase and squash (apologies to all if all my 'accept' generated insane spam)

grahamperrin

Thanks,

#202 (comment)

… Please let me know if I missed anything …

One typo (kenrel), so I took the opportunity to speed-read the whole thing with fresh eyes.

Take only what you want from the suggested changes, no need to explain any discards (unless you want to).

… if all my 'accept' generated insane spam)

It's a neat feature, I never thought of it as spammy :-)

website/content/en/status/report-2023-04-2023-06/kboot.adoc

grahamperrin · 2023-07-10T20:54:24Z

website/content/en/status/report-2023-04-2023-06/kboot.adoc

+It places them into memory as if it were running in a pre-boot environment and then loads that image into the Linux kernel with man:kexec_load[2] and does a special reboot to that kernel image.
+For UEFI enabled systems, it passes the UEFI memory table, and pointer to UEFI runtime services to the new kernel.
+
+It supports loading files from the host's filesystem, from any man:loader[8]-supported filesystem on the host's block devices (including pools that span multiple devices), from ram disk images and from files downloaded over the network.


Suggested change

It supports loading files from the host's filesystem, from any man:loader[8]-supported filesystem on the host's block devices (including pools that span multiple devices), from ram disk images and from files downloaded over the network.

It supports loading files from the local filesystem, from any man:loader[8]-supported filesystem on the host's block devices (including pools that span multiple devices), from ram disk images, and from remote filesystems.

I like your change. but would say 'from the network' rather than 'from networked files'. Comments?

Thanks. Suggestion revised. Have I twisted the meaning?

website/content/en/status/report-2023-04-2023-06/kboot.adoc

grahamperrin · 2023-07-10T21:06:15Z

website/content/en/status/report-2023-04-2023-06/kboot.adoc

+The amd64 boot environment places more requirements on the boot loader to provide data for the kenrel than aarch64 due to amd64 being an older port.
+All sources for data in the BIOS enviornment had to be provided by the boot loader since the kernel had no access to them from long mode.
+While UEFI and ACPI provide ways for the kernel to get this data, much of the data must still be provided by the boot loader.
+The kernel panics during initialization since all these prerequisites have not been discovered and implemented.


Suggested change

The kernel panics during initialization since all these prerequisites have not been discovered and implemented.

[.filename]#loader.kboot# does not yet implement the entire amd64 boot protocol, so the kernel still panics.

How about

Loader.kboot does not yet implement the entire amd64 boot protocol, so the kernel still panics.

Suggestion revised, NB:

lowercase l

[.filename]#⋯# markup seems reasonable here.

website/content/en/status/report-2023-04-2023-06/kboot.adoc

Co-authored-by: Graham Perrin <[email protected]>

bsdimp · 2023-07-10T21:33:07Z

Accepted all but two. What do you think of my suggested changes to your suggested change ... If you like them, just make them a suggestion and I'll accept it. If not, we'll chat :)

pauamma

Spelling and punctuation nits.

website/content/en/status/report-2023-04-2023-06/kboot.adoc

Co-authored-by: Pau Amma <[email protected]>

grahamperrin · 2023-07-10T22:39:00Z

website/content/en/status/report-2023-04-2023-06/kboot.adoc

+The normal FreeBSD [.filename]#/boot/loader# is built as a Linux binary that reads in the FreeBSD kernel, modules and tunables.
+It places them into memory as if it were running in a pre-boot environment, then loads that image into the Linux kernel with man:kexec_load[2] and does a special reboot to that image.


Suggested change

The normal FreeBSD [.filename]#/boot/loader# is built as a Linux binary that reads in the FreeBSD kernel, modules and tunables.

It places them into memory as if it were running in a pre-boot environment, then loads that image into the Linux kernel with man:kexec_load[2] and does a special reboot to that image.

The normal FreeBSD [.filename]#/boot/loader# is built as a Linux binary that reads in the FreeBSD kernel, modules, and tunables.

LinuxBoot places the FreeBSD kernel, modules, and tunables into memory as if it were running in a pre-boot environment, then loads that image into the Linux kernel with man:kexec_load[2] and performs a special reboot to the image.

Disambiguate the words 'It' and 'them' … correct, or have I misunderstood?

(I might be slaving over this unnecessarily; the report acknowledges the need for a manual page.)

as if loader.kboot were running as loader.efi and trying to load the kernel, tuneables and modules from there (similar memory layouts happen). I struggled with a way to say loader.kboot will present the same environment to the kernel's startup code after loader.kboot does its kexec that loader.efi does after loading everything and jumping to the kernel's startup code.

… I struggled with a way to say …

How about … a freehand sketch? (As if you have nothing better to do.)

It needn't be perfect; the absence of loader.kboot(8) is uppermost amongst your pleas for help.

File an image somewhere under /home/imp/public_html then it can be linked from within the report. Just a thought.

If an image is to be included, please remember that the report will also be sent by email. So either we include a link reference or maybe a picture in Ascii art.

But a good wording is probably better.

Co-authored-by: Pau Amma <[email protected]>

lsalvadore

I have found two last minute small issues.
I think we can finally merge this report, is anything still missing?

lsalvadore · 2023-07-16T10:06:17Z

website/content/en/status/report-2023-04-2023-06/kboot.adoc

+
+Help needed:
+
+ 1. man:loader.kboot[8] needs to be written. It should document how to use [.filename]#loader.kboot#, how to create images, and the use cases that work today.


Suggested change

1. man:loader.kboot[8] needs to be written. It should document how to use [.filename]#loader.kboot#, how to create images, and the use cases that work today.

1. man:loader.kboot[8] needs to be written.

It should document how to use [.filename]#loader.kboot#, how to create images, and the use cases that work today.

One sentence per line please, even in list elements.

Rolled into (superseded by) #202 (comment)

lsalvadore · 2023-07-16T10:06:29Z

website/content/en/status/report-2023-04-2023-06/kboot.adoc

+Unifying the kboot and efi copies is needed. +
+While they are mostly the same, sharing is complicated by remaining compile-time differences. +
+In addtion, the build infrastructure makes sharing awkward.
+ 1. It would be nice to add riscv64 support


Suggested change

1. It would be nice to add riscv64 support

1. It would be nice to add riscv64 support.

Rolled into (superseded by) #202 (comment)

grahamperrin · 2023-07-16T14:05:49Z

is anything still missing?

At a glance, at least:

substantially, kboot status report for 2023 Q2 #202 (comment), which, as revised, is Warner's suggestion
nelgigibly, where kboot status report for 2023 Q2 #202 (comment) begins to pay attention to an ordered list, now I can see a leading white space at each number.

https://docs.asciidoctor.org/asciidoc/latest/syntax-quick-reference/#lists example 31 presents what's below the lead for a simple ordered list:

.

grahamperrin · 2023-07-16T14:15:54Z

website/content/en/status/report-2023-04-2023-06/kboot.adoc

+ 1. The current elf arch-specific metadata code is copied from efi. +
+Unifying the kboot and efi copies is needed. +
+While they are mostly the same, sharing is complicated by remaining compile-time differences. +
+In addtion, the build infrastructure makes sharing awkward.


Is the hardness + of line breaks intentional?

Previewed:

grahamperrin · 2023-07-16T14:18:39Z

website/content/en/status/report-2023-04-2023-06/kboot.adoc

+PowerPC builds, but nothing more of its state is known.
+Attempts to acquire a suitable Playstation 3 proved to be too time consuming for the author.
+
+Help needed:


Suggested change

Help needed:

==== Help Needed

a simple subheading

and then, what's currently an ordered list of numbered items can become an unnumbered series of simple paragraphs.

The final paragraph, sponsorship already in place, will not be misinterpreted as something with which you need help.

lsalvadore · 2023-07-22T10:29:42Z

Thanks @bsdimp, @grahamperrin and @pauamma for your efforts to put in place a high quality report.

Do you think you can complete the last few things you need in the next few days? It is already late July, I would like to publish the status report within August 15th, or even better within the end of the month. It does not matter if it is not perfect, as long as the content is there.

If you do not have time and you want me to make the merge and fix any last minute thing, please just ask.

bsdimp · 2023-07-23T20:31:58Z

I thought that Iwas done. what's left?

bsdimp · 2023-07-23T20:34:04Z

oh, I see. The last few suggestions are fine: the hard breaks aren't intentional and the new section is fine. I don't have time to revise and repush, thought.

grahamperrin

Re: #202 (comment)

… repush …

If it simplifies things:

the two-click routine for each acceptable suggestion, then
https://patch-diff.githubusercontent.com/raw/freebsd/freebsd-doc/pull/202.diff should be ready for Lorenzo to push (if everything that's required existed as a suggested change for step 1).

grahamperrin · 2023-07-24T01:20:29Z

website/content/en/status/report-2023-04-2023-06/kboot.adoc

+ 1. man:loader.kboot[8] needs to be written. It should document how to use [.filename]#loader.kboot#, how to create images, and the use cases that work today.
+ 1. Finish amd64 support.
+ 1. The current elf arch-specific metadata code is copied from efi. +
+Unifying the kboot and efi copies is needed. +
+While they are mostly the same, sharing is complicated by remaining compile-time differences. +
+In addtion, the build infrastructure makes sharing awkward.
+ 1. It would be nice to add riscv64 support
+ 1. PowerPC testing (it has been untested since the refactoring started).
+ 1. Creating a script to repackage EDK-II image (say, from QEMU) as a linux-boot image with a Linux kernel built on FreeBSD for CI testing.
+ 1. Testing it from the coreboot LinuxBoot.


Suggested change

1. man:loader.kboot[8] needs to be written. It should document how to use [.filename]#loader.kboot#, how to create images, and the use cases that work today.

1. Finish amd64 support.

1. The current elf arch-specific metadata code is copied from efi. +

Unifying the kboot and efi copies is needed. +

While they are mostly the same, sharing is complicated by remaining compile-time differences. +

In addtion, the build infrastructure makes sharing awkward.

1. It would be nice to add riscv64 support

1. PowerPC testing (it has been untested since the refactoring started).

1. Creating a script to repackage EDK-II image (say, from QEMU) as a linux-boot image with a Linux kernel built on FreeBSD for CI testing.

1. Testing it from the coreboot LinuxBoot.

man:loader.kboot[8] needs to be written.

It should document how to use [.filename]#loader.kboot#, how to create images, and the use cases that work today.

Finish amd64 support.

The current elf arch-specific metadata code is copied from efi.

Unifying the kboot and efi copies is needed.

While they are mostly the same, sharing is complicated by remaining compile-time differences.

In addtion, the build infrastructure makes sharing awkward.

It would be nice to add riscv64 support.

PowerPC testing (it has been untested since the refactoring started).

Creating a script to repackage EDK-II image (say, from QEMU) as a linux-boot image with a Linux kernel built on FreeBSD for CI testing.

Testing it from the coreboot LinuxBoot.

consistent with Warner's kboot status report for 2023 Q2 #202 (comment)

consistent with (complementary to) my kboot status report for 2023 Q2 #202 (comment)

consistent with Lorenzo's per-line markup from kboot status report for 2023 Q2 #202 (comment)

includes the additional full stop from Lorenzo's kboot status report for 2023 Q2 #202 (comment)

seven paragraphs instead of points 1–7.

grahamperrin · 2023-07-24T01:51:33Z

My recent question to FreeBSD Git administrators <[email protected]> and FreeBSD Documentation Engineering Team <[email protected]> (addressees invisible at https://lists.freebsd.org/archives/freebsd-doc/2023-July/004013.html) was partly for people to learn whether a maintainer can perform actions such as:

clicking to accept a change that uses the suggested change feature of GitHub.

GitHub: allow edits by maintainers

Reviewed by: grahamperrin, status (Pau Amma <[email protected]>) Approved by: carlavilla (mentor, implicit) Pull Request: #202

lsalvadore · 2023-07-25T09:21:41Z

Merged, thanks.

I did my best to include the last minute changes that you agreed. If I misunderstood ot forgot something, please feel free to fix it, even without review (the month is coming to an end and I would like to speed things up to ensure the report is published within it).

I will close the PR as soon as I have access to a computer: for some reason, the GitHub app does not allow me to close PRs on my smartphone...

sergio-carlavilla · 2023-07-28T09:56:31Z

Merged.

github-actions bot added the status report label Jul 7, 2023

This comment was marked as resolved.

Sign in to view

This comment was marked as outdated.

Sign in to view

lsalvadore suggested changes Jul 9, 2023

View reviewed changes

This comment was marked as resolved.

Sign in to view

bsdimp force-pushed the kboot-2023q2 branch from 307edb8 to f530e2d Compare July 10, 2023 06:29

kboot status report for 2023 Q2

48ff45b

Sponsored by: Netflix

bsdimp force-pushed the kboot-2023q2 branch from f530e2d to 48ff45b Compare July 10, 2023 06:33

grahamperrin suggested changes Jul 10, 2023

View reviewed changes

bsdimp and others added 8 commits July 10, 2023 15:27

Update website/content/en/status/report-2023-04-2023-06/kboot.adoc

dd14635

Co-authored-by: Graham Perrin <[email protected]>

Update website/content/en/status/report-2023-04-2023-06/kboot.adoc

ed8da2b

Co-authored-by: Graham Perrin <[email protected]>

Update website/content/en/status/report-2023-04-2023-06/kboot.adoc

10aed4a

Co-authored-by: Graham Perrin <[email protected]>

Update website/content/en/status/report-2023-04-2023-06/kboot.adoc

142376b

Co-authored-by: Graham Perrin <[email protected]>

Update website/content/en/status/report-2023-04-2023-06/kboot.adoc

4ccc8a2

Co-authored-by: Graham Perrin <[email protected]>

Update website/content/en/status/report-2023-04-2023-06/kboot.adoc

084c392

Co-authored-by: Graham Perrin <[email protected]>

Update website/content/en/status/report-2023-04-2023-06/kboot.adoc

677ddca

Co-authored-by: Graham Perrin <[email protected]>

Update website/content/en/status/report-2023-04-2023-06/kboot.adoc

b9ba508

Co-authored-by: Graham Perrin <[email protected]>

pauamma reviewed Jul 10, 2023

View reviewed changes

website/content/en/status/report-2023-04-2023-06/kboot.adoc Outdated Show resolved Hide resolved

website/content/en/status/report-2023-04-2023-06/kboot.adoc Outdated Show resolved Hide resolved

Update website/content/en/status/report-2023-04-2023-06/kboot.adoc

1d28859

Co-authored-by: Pau Amma <[email protected]>

grahamperrin reviewed Jul 10, 2023

View reviewed changes

Update website/content/en/status/report-2023-04-2023-06/kboot.adoc

3ad7722

Co-authored-by: Pau Amma <[email protected]>

lsalvadore approved these changes Jul 16, 2023

View reviewed changes

grahamperrin suggested changes Jul 16, 2023

View reviewed changes

grahamperrin suggested changes Jul 24, 2023

View reviewed changes

freebsd-git pushed a commit that referenced this pull request Jul 25, 2023

Status/2023Q2/kboot.adoc: Add report

2ce9741

Reviewed by: grahamperrin, status (Pau Amma <[email protected]>) Approved by: carlavilla (mentor, implicit) Pull Request: #202

sergio-carlavilla closed this Jul 28, 2023


		Help needed:

		1. Currently, there is no documentation on how to use loader.kboot, nor on how to create test images.

	It supports loading files from the host's filesystem, from any man:loader[8]-supported filesystem on the host's block devices (including pools that span multiple devices), from ram disk images and from files downloaded over the network.
	It supports loading files from the local filesystem, from any man:loader[8]-supported filesystem on the host's block devices (including pools that span multiple devices), from ram disk images, and from remote filesystems.

	The kernel panics during initialization since all these prerequisites have not been discovered and implemented.
	[.filename]#loader.kboot# does not yet implement the entire amd64 boot protocol, so the kernel still panics.

		The normal FreeBSD [.filename]#/boot/loader# is built as a Linux binary that reads in the FreeBSD kernel, modules and tunables.
		It places them into memory as if it were running in a pre-boot environment, then loads that image into the Linux kernel with man:kexec_load[2] and does a special reboot to that image.


		Help needed:

		1. man:loader.kboot[8] needs to be written. It should document how to use [.filename]#loader.kboot#, how to create images, and the use cases that work today.

	1. man:loader.kboot[8] needs to be written. It should document how to use [.filename]#loader.kboot#, how to create images, and the use cases that work today.
	1. man:loader.kboot[8] needs to be written.
	It should document how to use [.filename]#loader.kboot#, how to create images, and the use cases that work today.

	1. It would be nice to add riscv64 support
	1. It would be nice to add riscv64 support.

kboot status report for 2023 Q2 #202

kboot status report for 2023 Q2 #202

Conversation

bsdimp commented Jul 7, 2023

This comment was marked as resolved.

bsdimp commented Jul 8, 2023

This comment was marked as outdated.

lsalvadore left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as resolved.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as outdated.

Choose a reason for hiding this comment

This comment was marked as resolved.

bsdimp Jul 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as outdated.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as outdated.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as resolved.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as outdated.

Choose a reason for hiding this comment

This comment was marked as resolved.

bsdimp commented Jul 10, 2023 • edited Loading

grahamperrin left a comment • edited Loading

Choose a reason for hiding this comment

grahamperrin Jul 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grahamperrin Jul 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bsdimp commented Jul 10, 2023

pauamma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lsalvadore left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grahamperrin commented Jul 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lsalvadore commented Jul 22, 2023

bsdimp commented Jul 23, 2023

bsdimp commented Jul 23, 2023

grahamperrin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grahamperrin commented Jul 24, 2023 • edited Loading

lsalvadore commented Jul 25, 2023

sergio-carlavilla commented Jul 28, 2023

bsdimp Jul 10, 2023 •

edited

Loading

bsdimp commented Jul 10, 2023 •

edited

Loading

grahamperrin left a comment •

edited

Loading

grahamperrin Jul 10, 2023 •

edited

Loading

grahamperrin Jul 10, 2023 •

edited

Loading

grahamperrin commented Jul 24, 2023 •

edited

Loading