RHEL8 with EFI after upgrade to 9.x doesn't boot, because of missed GRUB config file

[yuravk]

Actual behavior
System doesn't boot because of missed /boot/grub2/grub.cfg GRUB configuration file, after upgrading of recent Red Hat Enterprise Linux release 8.10 to 9.4, and if Extensible Firmware Interface (EFI) is used.

To Reproduce

1. Get system with most recent RHEL8 and EFI. Register it with RHSM, upgrade, disable zone drifting in the firewall.
2. Setup Leapp
3. Pre-upgrade check with Leapp, review results
4. Upgrade with leapp upgrade --debug
5. Reboot the system
6. Get into the system's main console, wait for Leapp to do the upgrade
7. After the upgrade, Leapp will reboot the system, and grub will enter its CLI because of missed configuration file (see screenshot attached)

Expected behaviour

Assuming a kind of "emergency shell", and chroot into the broken system

• OS version

[root@localhost-live /]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.4 (Plow)

• GRUB packages

[root@localhost-live /]# rpm -qa | grep grub2
grub2-common-2.06-80.el9_4.noarch
grub2-tools-minimal-2.06-80.el9_4.x86_64
grub2-tools-2.06-80.el9_4.x86_64
grub2-efi-x64-2.06-80.el9_4.x86_64
grub2-tools-efi-2.06-80.el9_4.x86_64
grub2-tools-extra-2.06-80.el9_4.x86_64

• GRUB's config file in case if EFI:

[root@localhost-live /]# ls -la /etc/grub2-efi.cfg
lrwxrwxrwx. 1 root root 22 May 30 19:20 /etc/grub2-efi.cfg -> ../boot/grub2/grub.cfg

points into the broken symlink:

[root@localhost-live etc]# ls -la ../boot/grub2/grub.cfg
ls: cannot access '../boot/grub2/grub.cfg': No such file or directory

[root@localhost-live /]# ls -la /boot/grub2/grub.cfg
ls: cannot access '/boot/grub2/grub.cfg': No such file or directory

Upgrade worked till Red Hat Enterprise Linux release 9.4 (Plow) and GRUB version 2.06-80.el9_4.

Leapp should correctly handle such configuration.

System information (please complete the following information):

• OS and version:

# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.10 (Ootpa)

• system information

# uname -a
Linux rhel8.tst 4.18.0-553.8.1.el8_10.x86_64 #1 SMP Fri Jun 14 03:19:37 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

• EFI configuration

# efibootmgr
BootCurrent: 0004
BootOrder: 0004,0000,0001,0002,0003
Boot0000* EFI Virtual disk (0.0)
Boot0001* EFI VMware Virtual SATA CDROM Drive (0.0)
Boot0002* EFI Network
Boot0003* EFI Internal Shell (Unsupported option)
Boot0004* Red Hat Enterprise Linux

• installed leapp packages

# rpm -qa "*leapp*"
python3-leapp-0.17.0-1.el8.noarch
leapp-upgrade-el8toel9-0.20.0-2.el8.noarch
leapp-deps-0.17.0-1.el8.noarch
leapp-0.17.0-1.el8.noarch
leapp-upgrade-el8toel9-deps-0.20.0-2.el8.noarch

• RHSM information

# subscription-manager list
+-------------------------------------------+
    Installed Product Status
+-------------------------------------------+
Product Name:   Red Hat Enterprise Linux for x86_64
Product ID:     479
Version:        8.10
Arch:           x86_64
Status:         Subscribed
Status Details: 
Starts:         xx/xx/xxxx
Ends:           xx/xx/xxxx

• GRUB configuration files on the system:

# ls -la /etc/grub2-efi.cfg
lrwxrwxrwx. 1 root root 31 Feb 21 15:12 /etc/grub2-efi.cfg -> ../boot/efi/EFI/redhat/grub.cfg
# ls -la /boot/efi/EFI/redhat/grub.cfg
-rwx------. 1 root root 6645 May 29 11:01 /boot/efi/EFI/redhat/grub.cfg

Attachments

leapp-logs.tar.gz

[pirat89]

hI @yuravk , this is actually bug in grub2 on rhel9.4. We are waiting for the fix.

[morete]

Hi @pirat89 , is there a jira at redhat that could be tracked for progress/status? I just lifted our upgrade path from 8.8 > 9.2 to 8.10 > 9.4 and I'm hitting this issue. I tried the proposed fixing actor in #1229 but it did not solve it... Thanks!

[pirat89]

@morete Hi. I just returned from vacation. yes, it is tracked here: https://issues.redhat.com/browse/RHEL-40362 however I see it has restricted access 🤷 . Here is related KCS article: https://access.redhat.com/solutions/7075830

We will leave here a new update when we have more info. Currently we have internally new grub2 build available for testing which should fix this behaviour. So far, fixed grub2 package is expected to be released earlier than leapp-repository, that's why we have not merged any changes in this repository yet.

Also the #1229 PR has been developed for a little bit different issue. Unfortunately, the fix in grub2 introduced a different problems for UEFI. That's why it's affecting now larger set of machines.

[pirat89]

We can confirm the problem is going to be fixed with the next grub2 9.4.z-stream release. Upgrades are passing with the new build as expected.

[morete]

@pirat89 thanks for the update.
Few weeks back I managed to see the EFI grub.cfg file after the failed upgrade is trying to source the old BIOS grub.cfg file location (which we no longer use, so does not exist) so I did workaround it by changing the automation code to create the grub config on both locations BIOS and EFI config file paths prior to running the leapp upgrade command and once the upgrade is finished then the automation reverts the workaround by deleting the old BIOS grub.cfg and updating the EFI grub.cfg... that made all work as expected so with that "temporal" workaround the upgrade works fine...

[pirat89]

@morete thanks for the info, yes that should be possible workaround for now.

[yuravk]

I can confirm the issue was fixed for me with grub2 2.06-82.el9 packages.

[pirat89]

The fixed grub2 build has been released \o/ So closing the issue. But note that we realized unfortunate thing, that if the machine has created both configs (the previously applied workaround for the original issue) it breaks the upgrade again. (@morete headsup in case it's still relevant for you)

[morete]

Hey @pirat89 , thanks for the heads up... Will keep this in mind... Thanks again!