Fix kexec framebuffer graphics for 5.10 kernel #1378
Conversation
kexec(8) needs to get the framebuffer address in order to set up the new kernel's boot parameters. This is one of the reasons that using a >4.20 kernel in Heads prevents framebuffer graphics from working in the OS kernel. Linux 4.20 started hiding this address from userspace, because userspace is not supposed to need physical memory addresses. A workaround was added to keep leaking the address, apparently for some proprietary userspace OpenGL drivers. This requires both a Kconfig and a kernel parameter. This commit enables the Kconfig on the librem_common config, and the kernel parameter on the librem_15v4 (where I'm testing this). We will need to enable it on other >4.20 configs/boards as well. Signed-off-by: Jonathon Hall <[email protected]>
The i915 driver's ID changed again, now to i915drmfb. It's unclear why kexec checks this, it seems it could populate the target kernel's framebuffer info as long as it knows enough about the host kernel's framebuffer, which it already checks. Maybe we could improve this, for now just add the new ID again. Signed-off-by: Jonathon Hall <[email protected]>
|
I suggest that we version bump kexec to latest. I added the following patch on my debugging branch so that we know what is happening when calling Might be a good idea to output fix.id content when no match happens into kexec. Otherwise we are kinda in the dark right now |
|
The big picture of how boot works is somewhat complex. In particular, it depends on whether coreboot provides any graphics init and a framebuffer that the kernel can observe (such as VBE). My boards don't provide anything from coreboot, and we would prefer not to have to include libgfxinit since we are already shipping a driver in the Heads kernel. qemu works differently and I have not investigated that yet, I'm focusing on hardware first. The boot is supposed to proceed this way:
The upshot of all this is that we can boot an OS needing a framebuffer without having to ship anything like a VBIOS, GOP driver, etc. We just use the Heads Linux kernel to set up a framebuffer. There's been a lot of discussion on this in a lot of places, and just to capture a few more details:
|
Agree, I'm working locally with 2.0.26, right now just want to get it functioning correctly and then we can expand with things like this.
Agree here too, I had to do the same to get the new-new ID. But maybe we can improve it not to check ID this way, since it's a bit of a cat-and-mouse game keeping up with the ID changes. Anyway, like above, I want to get it to fully work on 5.10, try other boards, and see what we can improve at that point. |
|
Here is my recent patch to also show output for newer board ports when fix.id doesn't match anything to proper into desired Unfortunately it doesn't help as of now for qemu + tinycore which still hangs there without neither serial nor vga, but shows on kexec -d -l call: But does't succeed kexec'ing into next kernel with vga... |
Compressed framebuffer requires the driver to track updates to the framebuffer from the CPU and update the compressed framebuffer. This doesn't work if we kexec into an OS that will use the linear framebuffer, so disable it. (The OS kernel can still use compressed framebuffer if it has i915.) Linux 5.8 enabled compressed framebuffer on more chipsets using i915, which is why this stopped working. memtest86+ and Debian (manually blacklisted i915, comparable to netinst) now boot correctly on Librem 15v4. This will need to be enabled for other boards too. Signed-off-by: Jonathon Hall <[email protected]>
|
As to be seen by the reference, this solution works for me on a novacustom nv4xpz (12th gen intel) together with some more twirks to get into heads initially (simplefb). Here's my current state: master...daringer:heads:nitropad-x Notable changes/differences: GOP init for gfx + simplefb + 6.1 kernel awesome work folks, thanks! |
|
b7adbb6 x230-maximized rom (https://output.circle-artifacts.com/output/job/fccd3747-19de-454c-ab0e-2341d1308ad4/artifacts/0/build/x86/x230-maximized/heads-x230-maximized-v0.2.0-1527-gb7adbb6.rom) was able to boot directly into Tinycore 64 bit (http://tinycorelinux.net/14.x/x86_64/release/TinyCorePure64-14.0.iso) dd'ed onto usb thumb drive. (@saber might be interested. Other isolinux fix are needed to boot detached signed iso, which is #1374 ) Will fix all other configs and push into seperate PR for cherry-picking. |
Was some configuration changed? If not, it’d be great, if you bisected this Linux kernel regression. |
|
Was Linux’ framebuffer configuration changed? Does building with simplefb fix it as it’s done in @daringer’s changes?
|
|
@paulmenzel I did bisect this (twice 😁 ) The first problem was this commit: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6be8f3bd2c78915a9f3a058a346ae93068d35c01 Linux now hides the framebuffer address from userspace. That makes sense, but kexec uses it to populate the boot parameters for the new kernel. There were a Kconfig and module parameter added later to keep the old behavior for compatibility, which I enabled here. The second problem (specific to i915 apparently) was this: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=691f7ba58d5220bbb06392054a6e50abdd393516 More devices started using a compressed framebuffer in more situations. We can only provide a flat framebuffer to the new OS, so I set the module parameter to disable this. I'm not really sure what the long term "perfect solution" is here, it seems like having kexec do this is making a lot of assumptions about how the kernel has set this framebuffer up, and it still doesn't work on ast (and probably others), looks like they are using a shadow framebuffer in the kernel and we need the real framebuffer address somehow. This works for i915 for now, but maybe a more ideal solution would be to have the host kernel provide its framebuffer address to the new kernel instead of doing it in userspace?
Don't think so, because coreboot isn't initializing graphics. The first graphics initialization is done by the Heads kernel's i915 driver (or other driver), then we use that as the "firmware" framebuffer for the new kernel.
This one is already enabled. |
Allow leaking the DRM framebuffer pointer to userspace, and disable framebuffer compression, like librem_15v4. Tested booting memtest86+ and Debian netinstaller on Mini v2. Do not enable this for L1UM, it uses Aspeed graphics which still don't work. qemu uses virtio graphics, which also are not working. Signed-off-by: Jonathon Hall <[email protected]>
Add CONFIG_DRM_FBDEV_LEAK_PHYS_SMEM and related kernel parameters to t440p. This board is already on kernel 5.10 and uses i915 graphics. Signed-off-by: Jonathon Hall <[email protected]>
For my hardware setup this was the only thing that helped for HEADS to display anything at all. But I did not try to leave it out and use your kernel commandline instead: @JonathonHall-Purism do you think using this command line instead of |
|
As a side note, I am using GOP init for the gfx and coreboot logs showed me this: which left me with the impression I need to activate FB somehow in the kernel, thus I activated |
|
@tlaurion I enabled the fixes for the other 5.10/i915 boards - other Librems except L1UM, and t440p. I tested mini v2 (I'll test the others in my next PB release). Could you test t440p? Aspeed graphics still do not work per the edited OP, I don't think I will be able to dive into this right away, I think this is worth finishing for i915 boards and merging. That won't be as simple (hah) of a fix since the real framebuffer address isn't plumbed near anywhere kexec could get it right now, as far as I can tell. I don't think we can drop the ID checks right now as a result. Maybe we could check if the smem_start is nonzero instead, not sure if ast is populating the shadow framebuffer here or nothing. I will still bump kexec to 2.0.26, this should be easy, and grab your improvements to tracing, particularly the driver name. |
It depends a bit on how graphics are initialized on your board. If it's like mine (nothing from coreboot, initialized by i915 in Heads kernel), then you need the kernel command line and config changes in this PR. CONFIG_SYSFB_SIMPLEFB will not help because there is no framebuffer provided by coreboot. If, on the other hand, you do have graphics initialized by coreboot, then you could try using simplefb and having kexec blindly reuse the video mode with --reuse-video-type. You'd also need to remove the graphics driver from the Heads kernel (i915 or other driver) so it doesn't change modes, which would invalidate the firmware-provided mode I believe. I don't really want to go that route on my boards though - I don't think libgfxinit supports them all so I would have to use VBIOS or a GOP driver (more blobs), and I'm not sure that would work on the servers (don't know if Aspeed's blobs would work without the full BIOS/UEFI environment). |
Update kexec to 2.0.26. Add tracing to framebuffer initialization. In particular, the driver name is traced if not recognized, and messages about kernel config are shown if the kernel doesn't provide the framebuffer pointer. Signed-off-by: Jonathon Hall <[email protected]>
JonathonHall-Purism
changed the title
|
@tlaurion Updated to kexec 2.0.26 and added tracing. I think I captured everything you had, and added a few things too. If there is anything that I missed that you think should be in here, let me know. Aspeed and qemu still aren't working but I don't really have bandwidth for another deep dive on this at the moment. There's no regression here, so I think the i915 work can be merged, no reason to hold it. |
|
@ThePlexus @srgrint @akunterkontrolle : I would need one of you confirming that:
|
|
I can confirm that I have flashed 353e836 on my t440p. The debian netinst from USB (I have tried bookworm test image) now boots fine. My installed debian distribution on the hard drive still works as well! |
|
This comment sounds like a blocker (thank you @lankredotp41 for mentioning it!): #1351 (comment) Per the comment, newer kexec now prefers multiboot2 by default, defeating the segment 0 / EBDA patch. We should check this out - @tlaurion the patch mentions Xen but I'm not sure if that whole chunk relates to Xen or just those few lines. Do you know offhand what this patch was for? |
We discussed it off github : coreboot savedefconfig format hides the facts, but mostly all librem boards currently enable libgfxinit and can now be more easily seen with helpers I added under 29c4b5c. The results above do not reflect the reality since my PR already attempts to fix some, but is not complete as of now. Ideal would really be to have Also note that boards relying on VGA_BIOS are out of the list: But that is no reason to not enable My only current doubt is around edp board including coreboot.review patch, which provides a new vbt and seems to rely on libgfxinit to do what otherwise seems to be done per FSP, but I would have to dive deeper into this, the simplest would be probably to also activate NO_GFX for that board as well and see how tings go gor board owners. |
|
Did someone test this with QubesOS, the same config which properly boots a recent Ubuntu is not working for me for QubesOS. It looks like the same issue we had with the linux kernel before. Using this https://github.com/daringer/heads/tree/nitropad-x setup... @marmarek are you maybe aware of issues with kexec with recent kernels + QubesOS/XEN ? |
rebased on top of this PR, pulled in missing patches - I can confirm that using the updated kexec leads to no issues. So booting an Ubuntu LTS looks good, QubesOS still doesn't work but I suspect a too old kernel version here |
|
@daringer as discussed off gihub: https://www.qubes-os.org/news/2023/03/14/qubes-4-1-2/#whats-new-in-qubes-412
You have tried this option on Q4.1.2 installer? |
Also discussed off github and above, kexec'ing into other kernel is not using the same codepath as multiboot (kexec 2.0.22) or multiboot2 (kexec 2.0.26). no-real-mode and vga=current are passed as additional arguments to Xen and kexec has a patch under heads tree that works for multiboot but not multiboot2 used to boot into xen. I have tested booting Qubes q4.1 and q4.2 from this PR on x230. But graphics only appears after OS loads i915 driver, so the kexec call seems "frozen" until Plymouth loads graphics (DRM+i915) just before the LUKS unlock passphrase prompt. Short version: i915drmfb additional I'd added into kexec patch affects non-multiboot kexec only: I have seen no improvement nor regression as compared to master when booting q4.1/q4.2(xen) whenever multiboot (master) or multiboot2 (this PR with kexec 2.0.26). |
|
I did some more testing on Mini v2 to clarify the behavior around coreboot libgfxinit. Bottom line, yes we can CONFIG_NO_GFX_INIT, and I'll set it on these boards once I test a few more devices. @tlaurion You're right that Librem boards actually did get libgfxinit by default, but the default is to initialize in text mode, so there was no framebuffer provided to the kernel like I had found earlier. Disabling i915 booted in text mode. CONFIG_X86_SYSFB (predecessor to CONFIG_SYSFB_SIMPLEFB) doesn't help either, which is not surprising since the output is in text mode. I tried configuring coreboot to initialize in framebuffer mode instead (CONFIG_GENERIC_LINEAR_FRAMEBUFFER) but this seems to just fail on this board, the video output was not enabled at all. There was no obvious tracing in coreboot indicating any error. I didn't look into it further since this is not the direction we want to go anyway. I'll test Qubes too. |
|
Tested 13v2, 14, and Mini v2, memtest86+ and PureOS work. I can reproduce the issue with Qubes though; installed 4.1.2 on my L14 again and get no video on boot. I'll need to investigate what is going on there. |
We don't need coreboot to initialize graphics on this boards, this eliminates some unneeded code and the gnat dependency for them. Coreboot was using libgfxinit, but it was initializing in text mode. Heads' kernel will then switch to graphics mode, and we hand that framebuffer from i915 to the target kernel during kexec. Signed-off-by: Jonathon Hall <[email protected]>
|
Well, I no longer have the issue with Qubes, not sure what was causing it. I set it up on Mini v2 to get serial logs. First boot after set up appeared to have no video output, but after about 5 minutes it actually did bring up video with i915. After that every boot has been fine, and I tried installing again to see if it was a first-boot issue, that was fine too. The install that was on my L14 is fine now too. (Still only have graphics after i915 init, but that's the same as master.) Unfortunately I didn't get a chance to collect any diagnostics since I wasn't expecting it to disappear on its own. Given the circumstances, this may not be introduced by this PR. @daringer Are you still having issues booting Qubes? |
I will clean up my working state and PR it here, this will then be easier to (and a better place to) discuss this. |
|
On Thu, Apr 27, 2023 at 02:19:31PM -0700, Markus Meissner wrote:
Yes, I still do. Are you using exactly this PR? I am afraid my problem might be a too old kernel version. But you say you installed QubesOS, already this is not working for me, once I kexec into the usb-device with the QubesOS image my display doesn't show anything.
Are you doing kexec with -latest kernel as dom0, or the default one?
…--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
|
Nope, I didn't - or to be more precise: I thought I did. Anyways it now works, the issue was self-inflicted as we removed the 6.1-kernel option from our QubesOS OEM images. Shot my own feet, both at once 🤓 @JonathonHall-Purism so I can confirm that there are no issues with QubesOS for this PR |
@marmarek do you recall tests where qubesos would give early kernel console output prior if DRM driver kicking in? |
Not really. That might have been the case before (Qubes 4.0?), but nor my memory nor test logs span that long to confirm/deny. |
|
Haven't finished pushing 5.x config changes to xx20/xx30 yet but since they aren't impacted as of today, not a real problem |
|
Can confirm as well for the t440p that this pull request fixes the issues with no framebuffer graphical output. More detailed comment: #1323 (comment) |
Fix setting up a framebuffer for the OS kernel to use following kexec.
If Heads is configured with kernel 4.20 or later, the OS kernel is not able to use framebuffer video output. The OS kernel must have the i915 module (or other appropriate driver) and there won't be any video output until it takes over the device. In particular, this prevents booting memtest86+ and the Debian netinstall image (as well as others).
This now works on i915 using kernel 5.10 in Heads. I've tested Librem 15v4 and Librem Mini v2.
It does not work on Aspeed graphics (ast), even when adding the astdrmfb identifier to kexec. It looks like ast uses a shadow buffer internally in the kernel (it sets
mode_config.prefer_shadow = 1in ast_mode.c), which means that the generic DRM FB helper doesn't know where the real framebuffer is (AFAICT from a cursory read-through). A hack would be to somehow get the real framebuffer address up to fix.smem_start, but this is pretty wrong from the kernel's perspective (nothing should write there, it's just what we need since we are going to kexec and the driver will be gone), would just be a hack for kexec. I took an initial look at this but it's going to need a deeper dive.That's probably the reason for the ID checks in kexec, so I don't think we can remove that currently.
TODO: