FreeBSD support #673
Replies: 27 comments 14 replies
-
|
FreeBSD booting has been a long-standing project "want" - we'd absolutely welcome it! Roughly speaking, how would the |
Beta Was this translation helpful? Give feedback.
-
|
+1 for this. I've dreamt of FreeBSD support ever since the OpenZFS unification. |
Beta Was this translation helpful? Give feedback.
-
|
To boot entirely off a ZFS BE, the process would be approximately
Setting 'bootdev' like this will cause the boot loader to use that as the 'load' device for both the loader's file (lua scripts, loader.conf) as well as finding the kernel here. Both will also clear 'bootonce' if it is set, but the latter will boot the geribils BE regardless of its current bootonce setting. For this project, the second form is likely preferable (the first form is actually loader.kboot's default). There's also ways to pass in a mfs image, should that be desirable. We use with a loader.conf in that has image that has in it. loader.kboot has a number heuristics to guess the right thing to use, so it sees that the boot device should be this MD file. It's unclear how helpful these probing things are, and you may need to be more explicit with |
Beta Was this translation helpful? Give feedback.
-
|
A few questions:
ZFSBootMenu is modular enough / we already have OS detection via |
Beta Was this translation helpful? Give feedback.
-
I'll have to see how hard it is to build on Linux... I have a linux test machine that was just built for me that I need to log into for amd64 kboot work anyway... /stand is built such that the loader.kboot is a linux binary, no matter what... Some of the other loaders may be less tolerant of that, but 'can be built' might mean needing to build a few things piecemeal, then linking loader.kboot.... we do pull from bits of the FreeBSD kernel, OpenZFS sources, FreeBSD's libc, etc when building the boot loader stuff, so it would be a relatively full source tree... |
Beta Was this translation helpful? Give feedback.
-
|
Any updates on this issue? 😄 |
Beta Was this translation helpful? Give feedback.
-
|
Bumping this up for attention. Is there any help needed to kickstart the work on this ? |
Beta Was this translation helpful? Give feedback.
-
|
I have LinuxBoot working for FreeBSD/arm64, but not for FreeBSD/amd64 yet. I guess, what do you need from me to proceed? |
Beta Was this translation helpful? Give feedback.
-
Unfortunately, right now we realistically only support amd64. The most readily available arm64 hardware - RPI - has a number of problems that limits it's usefulness for us. Once amd64 support is available, we'll be happy to start hacking on integrating that tool into ZBM! |
Beta Was this translation helpful? Give feedback.
-
|
Well, FreeBSD/amd64 loader.kboot builds and runs on the Linux side, FreeBSD has an annoying couple of bugs still that panic the boot... I'll let you know when that changes. |
Beta Was this translation helpful? Give feedback.
-
|
hello, I'm looking into this. have you discovered anything new since last month? |
Beta Was this translation helpful? Give feedback.
-
|
I still need to fix a few things on the amd64 side of the house for LinuxBoot, and $WORK has had me focused on other stuff. Though soon I should become focused on it again now that we're entering an end of the year quiet period. |
Beta Was this translation helpful? Give feedback.
-
|
Nice to meet you, bsdimp. I got the kboot building on my gentoo host, nice work by the way. |
Beta Was this translation helpful? Give feedback.
-
|
Sure. The more the better. aarch64 works fairly well (though I have had some problems pushing the linuxboot linux kernel and inintrd all the way into the UEFI FD image). |
Beta Was this translation helpful? Give feedback.
-
|
@philmb3487 and by 'kboot' do you mean the loader.kboot that's in FreeBSD? |
Beta Was this translation helpful? Give feedback.
-
yes absolutely, everything in stand/kboot. Not fully up to date with everything, but I'm getting more familiar with how it works by the minute, I suppose. is this an error in the code generation you think? |
Beta Was this translation helpful? Give feedback.
-
|
So I think maybe you misunderstand what loader.kboot is supposed to do... It's a Linux binary that is run instead of kexec to load the FreeBSD kernel. Unlike other places where loader runs (well, except bhyveload), this is inside the context of an OS, not bare metal. A large part of the point of doing that is to leverage the Linux kernel's ability to read and write filesystems on different devices that the boot loader might otherwise have to know how to do. |
Beta Was this translation helpful? Give feedback.
-
|
I think the problem is with identity mapping. There's something wrong with the page table. See the pictures, in the first, the kexec system call is about to fire. In the second, we're in the new kernel location, with a .text that is complete garbage. Perhaps it's not identity mapped here, to go PA->VA there's a translation step, something like BASE + PAGE_OFFSET (this is the high memory address I think? 'va' for the kernel is in high ram). Maybe nothing I said makes sense, patience with me please as this noob learns. |
Beta Was this translation helpful? Give feedback.
-
|
so what's the full call trace? And what's the error that kexec_load is returning? |
Beta Was this translation helpful? Give feedback.
-
|
Update, I have it loading the FreeBSD kernel, and booting into early init code. But there is a problem in the lgdt() load in machdep.c:954, as the stack seems to be off by a few bytes, and the machine hangs up. |
Beta Was this translation helpful? Give feedback.
-
|
Holy, I kexec'd into FreeBSD! It's pretty cool for sure. I wrote a little utility 'kexecbsd' to boot freebsd kernel's. I need to add support for AMD64. At the very least, now I know it can be done, and how. |
Beta Was this translation helpful? Give feedback.
-
|
I'm struggling with AMD64. It's a lot stricter when it comes to memory accesses, I'm getting a general fault when using any sort of 'push' instruction and pulling my hair out trying to figure it out. My GDT allows writes. what could make 'push' break? |
Beta Was this translation helpful? Give feedback.
-
OK you won't believe how dumb this one was, it turns out GDB commands 'nexti' and 'stepi' behave very differently, and I was using the wrong command. 'stepi' does what I want, but 'nexti' got the debugger very confused thinking there's a subroutine to forward and everything crashed. |
Beta Was this translation helpful? Give feedback.
-
|
I'm spending more time on this. Trampoline entry ends up at 0x0000000012b00000, but the loader seems to think it should go to 0x12d00000. Paging structure PML4T should be located at 0x12d01000, but that location is not filled in properly (there's just ASCII text, the actual mapping code looks OK). |
Beta Was this translation helpful? Give feedback.
-
|
That first segment looks suspicious, that for sure should not be there. Also, I'd like to add a column 'NAME' so we know what's being loaded. Maybe that's not possible if printing the kexec_segment structure... MEMSZ looks OK. |
Beta Was this translation helpful? Give feedback.
-
|
Yea, last time I was actively doing things, we'd panic. But we do store the page table after the trampoline page. And there's an offset sometimes in reporting. As luck would have it, linuxboot is next up on my work dance card, which I start Monday after a trip to the bay area. Maybe I'll try it on my mac + qemu so I can use it on the trip this week instead. I kept hitting issues with amd64 when I first started doing the work, so I switched to arm64, and we routinely boot FreeBSD with kexec on an embedded platform we're working to integrate with, so I know it works well on arm64. Now to get it to work for amd64... Do you have any FreeBSD changes to get to where you are? |
Beta Was this translation helpful? Give feedback.
-
Yeah, the paging is fine. My toy experimental loader booted fine with that algorithm. Set the physical memory below 4GB identity mapped, and the top 2GB of RAM to the loaded kernel. And the kernel started at ffffffff803d7000.
I wish you good luck then. Please let your boss know I would be more than willing to work on whatever project they have, should they be okay with a remote worker... :-)
space_avail() seems to be the culprit, as far as I can see. By faking the available memory, Segment 1 is clearly the loaded ELF bundle, with modules, and segment 2 is clearly the trampoline code, along with the 9 paging tables that go in cr3. I have not tested yet, that should come tomorrow, but it's looking better. Here are my recommendations, parsing /proc/iomem is fine, but there's an even easier way, the /sys filesystem on linux provides firmware/memmap/ for information about the memory segments. But never mind, the /proc/iomem way should be fine, just a bit more parsing to do. The seg.c file does the right thing, I think, but it doesn't seem to be called properly, because otherwise the first segment would load properly. In other words, I'm talking about, space_avail() should panic in case 0 is the result. I think of it as allocating memory, in case allocating memory for the kernel fails, there's really nothing we can do to recover. I think there's no need to go so high.... I copied the kernel in memory to 0x200000 (as is indicated in the elf file, PA field of the program's LOAD entries) and that worked just fine. The machine should pretty much always have memory at 2MiB we can use. Perhaps there are EFI considerations here I'm not thinking about, that seems to be the reason for "staging", ie you're copying around twice the kernel? |
Beta Was this translation helpful? Give feedback.
-
|
@bsdimp Doh, the code already reads from /sys/firmware/memmap, so that's great. Looks like I read too fast. Here's a patch that hopefully gets rid of the gnat with space_avail(), --- a/stand/kboot/kboot/arch/amd64/load_addr.c
+++ b/stand/kboot/kboot/arch/amd64/load_addr.c
@@ -53,11 +53,11 @@ struct kv
{ system_ram, "System RAM" },
{ acpi_tables, "ACPI Tables" },
{ acpi_nv_storage, "ACPI Non-volatile Storage" },
- { unusable, "Unusable memory" },
+ { unusable, "Unusable memory" },
{ persistent_old, "Persistent Memory (legacy)" },
{ persistent, "Persistent Memory" },
{ soft_reserved, "Soft Reserved" },
- { reserved, "reserved" },
+ { reserved, "Reserved" },
{ 0, NULL },
}; |
Beta Was this translation helpful? Give feedback.
-
|
I'm finally back to this full time, and this doesn't do the job for me. I notice that the unusable line is just whitespace. Is that intentional? |
Beta Was this translation helpful? Give feedback.
-
|
We're parsing it, but we're not doing it correctly. As part of doing the arm64 work after the initial amd64 support I refactored this stuff. I need to redo the amd64 stuff with that, and I'll likely just grab /sys/firmware/efi/runtime-map... unless you've already done that :). |
Beta Was this translation helpful? Give feedback.
-
Here is my patch for the day, diff --git a/stand/kboot/kboot/arch/amd64/load_addr.c b/stand/kboot/kboot/arch/amd64/load_addr.c
index 4bd2a19dab48..f02ec1488625 100644
--- a/stand/kboot/kboot/arch/amd64/load_addr.c
+++ b/stand/kboot/kboot/arch/amd64/load_addr.c
@@ -57,7 +57,7 @@ struct kv
{ persistent_old, "Persistent Memory (legacy)" },
{ persistent, "Persistent Memory" },
{ soft_reserved, "Soft Reserved" },
- { reserved, "reserved" },
+ { reserved, "Reserved" },
{ 0, NULL },
};
@@ -102,6 +102,13 @@ enumerate_memory_arch(void)
}
nr_seg = n;
+ if (!nr_seg) {
+ printf("Could not obtain memory tables, expect failure\n");
+ }
+
+ populate_avail_from_iomem();
+
+ print_avail();
return true;
}
diff --git a/stand/kboot/kboot/seg.c b/stand/kboot/kboot/seg.c
index 558cac501224..12991cd9c93f 100644
--- a/stand/kboot/kboot/seg.c
+++ b/stand/kboot/kboot/seg.c
@@ -205,20 +205,6 @@ enum types {
unknown,
};
-static struct kv
-{
- uint64_t type;
- char * name;
- int flags;
-#define KV_KEEPER 1
-} str2type_kv[] = {
- { linux_code, "Kernel code", KV_KEEPER },
- { linux_data, "Kernel data", KV_KEEPER },
- { linux_bss, "Kernel bss", KV_KEEPER },
- { firmware_reserved, "reserved" },
- { 0, NULL },
-};
-
static const char *
parse_line(const char *line, uint64_t *startp, uint64_t *endp)
{
@@ -251,16 +237,6 @@ parse_line(const char *line, uint64_t *startp, uint64_t *endp)
return (walker + 3);
}
-static struct kv *
-kvlookup(const char *str, struct kv *kvs, size_t nkv)
-{
- for (int i = 0; i < nkv; i++)
- if (strcmp(kvs[i].name, str) == 0)
- return (&kvs[i]);
-
- return (NULL);
-}
-
/* Trim trailing whitespace */
static void
chop(char *line)
@@ -272,7 +248,7 @@ chop(char *line)
}
#define SYSTEM_RAM_STR "System RAM"
-#define RESERVED "reserved"
+#define RESERVED "Reserved"
bool
populate_avail_from_iomem(void)
@@ -281,7 +257,6 @@ populate_avail_from_iomem(void)
char buf[128];
const char *str;
uint64_t start, end;
- struct kv *kv;
fd = open("host:/proc/iomem", O_RDONLY);
if (fd == -1) {
@@ -316,28 +291,6 @@ populate_avail_from_iomem(void)
add_avail(start, end, firmware_reserved);
else
goto next_line; /* Ignore hardware */
- while (fgetstr(buf, sizeof(buf), fd) >= 0 && buf[0] == ' ') {
- chop(buf);
- str = parse_line(buf, &start, &end);
- if (str == NULL)
- break;
- kv = kvlookup(str, str2type_kv, nitems(str2type_kv));
- if (kv == NULL) /* failsafe for new types: igonre */
- remove_avail(start, end, unknown);
- else if ((kv->flags & KV_KEEPER) == 0)
- remove_avail(start, end, kv->type);
- /* Else no need to adjust since it's a keeper */
- }
-
- /*
- * if buf[0] == ' ' then we know that the fgetstr failed and we
- * should break. Otherwise fgetstr succeeded and we have a
- * buffer we need to examine for being a top level item.
- */
- if (buf[0] == ' ')
- break;
- chop(buf);
- continue; /* buf has next top level line to parse */
next_line:
if (fgetstr(buf, sizeof(buf), fd) < 0)
break;
@@ -364,5 +317,5 @@ space_avail(uint64_t start)
* Properly used, we should never get here. Unsure if this should be a
* panic or not.
*/
- return 0;
+ panic("No memory available");
}Good thinking on using /sys/firmware/efi/runtime-map.... In the patch above, I simplify and directly use the memory regions that are in the system. I don't think it's necessary to protect the kernel memory, the kexec copies buffers after the kernel shuts down, so there's no issue there. Well, now what's happening is that the kexec jumps to (trampcode = 0x300000) which happens to be nonsense, and then the whole thing reboots. No idea why that location is chosen, that will have to wait tomorrow. |
Beta Was this translation helpful? Give feedback.
-
|
So as you're discovering, we're not putting the parsed data in to avail from seg.c. This is because I rewrote all the parsing to be more generic in the aarch64 port, but didn't spend time refactoring amd64 to use it, so there's no way the current code can work. I'm doing that refactoring now. One thing that makes it a little hard is that I'd like to populate the seg table from the EFI map, but I had tried to make kboot work on systems that didn't allow reading from RAM or even opening /dev/mem. Those system I would read EFI memory map in the trampoline. I may still support them (since I have the fallback code for iomem already, I may just keep it and write the populate from EFI if we have it and adjust the aarch64 stuff). I may be to the point of posting a branch in a few days to at least get the kernel loading and starting and hitting the next round of problems I know linger there. |
Beta Was this translation helpful? Give feedback.
-
|
OK i don't want to step on your toes, I'll wait for that refactor and then test it. |
Beta Was this translation helpful? Give feedback.
-
|
Can you explain what 'staging' is, and where it's used? |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately, some of the data needed is in the PA that we want to read. And aarch64 is deficient in not exporting it, at least in the kernels that I have, the way that x86_64 does. The notion that a 'secure kernel' is laughable though with one that can kexec and literally write arbitrary code to be run, though it does increase the chance of being noticed when you do the thing to get the data. I'm highly skeptical about it going away completely. But I dislike clever hacks like this to get around this information hiding when it's not information that should be hidden (so that cuts both ways: I shouldn't have to go to PA to get this data, but Linux's lack of uniform interfaces forces my hand). Also unfortunate, is that the EFI memory table isn't a useful thing to use to place the kernel. There's bits of 'conventional ram' that are still in use that can't be overwritten with the new kernel during the loading process. So maybe I'm arguing a moot point :). I think in this context 'staging' is just the base address of the memory arena we use. At least if I understand the question. Also, Reserved vs reserved I think didn't matter for the amd64 system, but would for the aarch system. It also might have been some stupid thing I did during the initial switch to the 'avail' code that broke amd64 before I broke it in other ways. It looks like the EFI memory table will need to be harvested in different ways, but maybe we can use /proc/iomem everywhere to get reserved data. I'll have to see if the /sys/firmware sources include or exclude the kernel in-use memory (if it doesn't, then my earlier experiments just got lucky). |
Beta Was this translation helpful? Give feedback.
-
|
So kexec / EFI / Linux question. On aarch64, I can easily get the EFI systbl (EFI_SYSTEM_TABLE) and from that get both the config table and the config table length. I think I have that covered (with our without /dev/mem, the only casualty of no /dev/mem is printing the table in the loader). But on amd64/x86_64, I have only pointers to the config table, the fw_vendor and the runtime services table. There's nothing that points to the systbl nor does there seem to be a way to get the number of entries in the config table. I'm having trouble finding where this comes from in the kexec case. The efi_setup_data has a PA for the tables (as element tables), for smbios, for the runtime table, etc. But not the count of number of entries. I don't see anything in the kernel that 'probes' for that, nor any math done to get/set nr_tables. I don't see it doing something weird as finding the right map entry and inferring the length from that... Anybody here know what I'm missing (or know who to ask to wisper the right file to read or the right routine to grep for)? All the places inside the Linux kernel fish it out of systab, but I don't see where it's set in the kexec path since kexec can't get systab directly, only the fields that are valid after ExitBootServices, except nr_tables... I'm feeling extra unobservant today. |
Beta Was this translation helpful? Give feedback.
-
|
So I think what's going on:
This means that the data from (2) isn't exported via sysfs because purgatory, the in-kernel-kexec-loader copies it over. The pointer to the boot_param structure is stored in rsi. So that means we'll have to snag systbl from the efi_info in the trampoline (offset 0x1c0 from %rsi) and write it to an address we burn into the trampoline code which corresponds to the metadata section we associate with the kernel. And we'll also need a new section that we'll pass in the PAs we harvest from the sysfs for runtime, cfgtbl and fw_vendor. This will let the kernel piece together the systbl, and the PAs it needs to access the runtime and cfgtbl stuff (since the pointers stored there can no longer be assumed to be 1:1 PA == VA since SetVirtualAddresses rewrites them). I'll have to create a new metadata section for that which the kernel can read and cope in the kexec case. We can reuse a lot of the code that I did for aarch64 which provides the PA if the systbl in a sensible way. Not sure why the same issue isn't there. So, once all this is coded up, I should be passing in all the right EFI data for us to make good progress, though the VA != PA issues will likely need some more intensive FreeBSD kernel hacking than I wanted to do if I want to support EFI variables (which i want to because $WORK is dependent on it). So that's how Linux gets the length of the cfgtbl.... Simple and clear, eh? Oh, and arch/x86/include/uapi/asm/bootparam.h and arch/x86/kernel/kexec-bzimage64.c are the breadcrumbs I'll need in the future. |
Beta Was this translation helpful? Give feedback.
-
|
Sounds good. I don't know too much about UEFI services, I've just booted the kernel using simple PC BIOS method for now, so I can't comment on this. In general I would expect Linux does its own table detection, so you don't need to pass in that data. But idk. |
Beta Was this translation helpful? Give feedback.
-
|
I'd be interested to see the patches that got you this far.... EFI vars are a nice to have, as are a few other bits of data from EFI. I've found how the linux kernel does the UEFI handoff and written some code to cope.. i got a little distracted into printing tracebacks on crashes. |
Beta Was this translation helpful? Give feedback.
-
|
Yes of course. Use as you wish. I have to find work, so this is going on pause for me. |
Beta Was this translation helpful? Give feedback.
-
|
Great! Thanks! I finally have my 'tracekback' code written, along with all the weird detours and work issues (I work for Netflix, and so have a problem or two to hunt) |
Beta Was this translation helpful? Give feedback.
-
FreeBSD now has loader.kboot for booting from Linux into FreeBSD. this works really well on aarch64, and will soon work well for amd64 (once I find the time to finish it).
I've been asked if ZBM could support FreeBSD booting. Looking quickly at the code, it looks like that could be accommodated with a custom detection for FreeBSD and a different boot recipe using loader.kboot instead of the kexec tools that the current script uses.
Would you be open to working together to make this happen?
Beta Was this translation helpful? Give feedback.
All reactions