[Raspberry Pi5] bootloader doesn't detect nvme WD_Black SN850X

[Jixel]

Describe the bug

ISSUE:
RaspBerry Pi5 8G + Hat Geekworm X1015 + nvme WD_Black SN850X 2Tb PCIe 4.0.
Bootloader doesn't detect nvme WD_Black SN850X

TEST:
RaspBerry Pi5 8G + Hat Geekworm X1015 + nvme SABRENT Rocket Nano 2242 512Gb PCIe 3.0
Bootloader works fine and detect and run Raspberry Pi OS

Steps to reproduce the behaviour

sudo apt update
sudo apt upgrade
sudo rpi-eeprom-update -a

BOOT_ORDER=0xf146
PCIE_PROBE=1

config.txt
dtparam=nvme
dtparam=pciex1_gen=3

Device (s)

Raspberry Pi 5

Bootloader configuration.

BOOT_UART=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf146
PCIE_PROBE=1

System

EEPROM

BOOTLOADER: up to date
CURRENT: lun 23 sep 2024 13:02:56 UTC (1727096576)
LATEST: lun 23 sep 2024 13:02:56 UTC (1727096576)
RELEASE: latest (/lib/firmware/raspberrypi/bootloader-2712/latest)
Use raspi-config to change the release.

config.txt

dtparam=nvme
dtparam=pciex1_gen=3

Bootloader logs

No response

USB boot

No response

NVMe boot

No response

Network (TFTP boot)

No response

[timg236]

I'm pretty SN850X PCIe 4 devices don't work at GEN2 and are incompatible, please check that it works under Linux. If not it's not going to work as a boot device.

[Jixel]

If I remove the Hat and connect SN850X via USB with a pcie/usb adapter, rpi5 bootloader runs correctly from USB and rpi5 works fine.
I have read that the latest version of pieeprom has introduced compatibility with the WD SN350, SN550 and SN580 series. Could this be a potential solution?

https://wiki.geekworm.com/NVMe_SSD_boot_with_the_Raspberry_Pi_5

[timg236]

To add more detail to my previous comment the question is does NVMe work natively under Linux when the Pi is booted from SD i.e. can you mount the drive, copy files, run dd tests etc.

If a USB adapter is added then the NVMe is talking to a PCIe interface inside the adapter and not the BCM2712 PCIe RC. Linux now just sees a USB-MSD device with some flash memory.

If you want to try a more recent bootloader then flash RPi OS to a spare SD-card, run "sudo rpi-update" then check the native NVMe behavior. There are some improvements for timing and switches.

However, there are some WD Black devices which only work at Gen4 and if these don't work natively from Linux then they definitely aren't going to work with the bootloader.

[Jixel]

Hi timg236,
Folllowing your steps:

• Format nvme to exFat.
• Connect SN850X by Pcie (Hat) to rpi5.
• Use microSD to boot rpi5.
• Rpi5 runs, detects nvme and I mount it.

• Copy files ok.

• Use "SD card copy" again for copying OS from microSD to nvme (Finished ok).
• Config boot order.
• Config config.txt.
• Shutdown rpi5, remove microSD and boot.

But... same error, rpi5 can't boot from WD_Black SN850X.
Any other suggestions?

Thanks a lots

[microfx]

I am in the same boat... my pineberry HatDrive! Bottom 2024/v5 won't let Raspberry Pi OS detect my WD 850X. I tried it all by now...

[pelwell]

@Jixel's log (in a screenshot, so not exactly searchable) shows an unhappy NVME interface with various error messages. Booting from an SD card but with the NVME drive attached, what does dmesg | grep nvme report for you?

[microfx]

dmesg | grep nvme
[    0.000000] Kernel command line: reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe  smsc95xx.macaddr=D8:3A:DD:D7:FD:85 vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000  console=tty1 root=PARTUUID=5a7c6c55-02 rootfstype=ext4 fsck.repair=yes rootwait cfg80211.ieee80211_regdom=DE nvme_core.default_ps_max_latency_us=2500

...so basically returns only a desperate attempt of trying everything I found on the interweb. I think I will bring back this SSD to the shop as my Samsung "just works" with the latest gen HatDrive! ...

[stewartoallen]

I just got a new pi5 + M.2 Hat+ today with this same exact drive and same error. however, I can boot from SD and access this drive normally when connected to the hat. so the drive does work. just not for boot.

I tried Geerling's old guide which uses rpi-clone and that got quit a bit further into the boot, but ended up in the same spot with a timeout waiting for the drive to show up.

[NeddySeagoon]

Team,

I may be able to add a little here. I don't know what logs may be useful, so I'll wait for some pointers.
With the boot order set to

Pi CM5 ~ $ rpi-eeprom-config 
[all]
BOOT_UART=1
# Switch off PMIC outputs on HALT
POWER_OFF_ON_HALT=1

# PCIE_PROBE=1 should not be needed
PCIE_PROBE=1

# Default BOOT_ORDER for provisioning
# SD -> NVMe -> USB -> Network
# BOOT_ORDER=0xf2461

# We want USB before NVME
BOOT_ORDER=0xf2641

and identical kernels installed on the nvme and a USB stick

0000:01:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Black SN850X NVMe SSD [15b7:5030] (rev 01)
	Subsystem: Sandisk Corp WD Black SN850X NVMe SSD [15b7:5030]
	Kernel driver in use: nvme

booting the USB stick, its before nvme in the boot order, gets me

[    0.336229] brcm-pcie 1000110000.pcie: Forcing gen 2
[    0.437667] brcm-pcie 1000110000.pcie: clkreq-mode set to default
[    0.437989] brcm-pcie 1000110000.pcie: link up, 5.0 GT/s PCIe x1 (!SSC)

which is what we want to see and working /dev/nvme* nodes.

Booting from the nvme, by unplugging the USB stick, gets me link down and no /dev/nvme* nodes.

The kernel is

$ uname -a
Linux Pi CM5 6.12.6-v8-16k+ #6 SMP PREEMPT Sun Dec 22 18:20:37 UTC 2024 aarch64 GNU/Linux

Which is the default bcm2712 with Device Mapper built in rather than a module, so I can do root in LVM with a module free initrd.
I've been told that doesn't work but I was doing it for about a year on a Pi5 before I heard that.

Once the nvme is up, I can chroot into the install and it works as expected.
That's boot an install from the USB stick.
vgchange -ay to get the logical volumes visible.
Mount all the bits at /mnt/gentoo
chroot /mnt/gentoo /bin/bash

In the interests of full disclosure, I'm a Gentoo person since 2003

[timg236]

It might be worth trying the latest bootloader (2025-01-08). This initializes PCIe PWR before DDR init giving the NVMe controller firmware at least two seconds to boot before PCIe enumeration.

[NeddySeagoon]

timg236,

Thank you. I now have

(chroot) Pi CM5 / # rpi-eeprom-update
BOOTLOADER: up to date
   CURRENT: Wed  8 Jan 17:52:48 UTC 2025 (1736358768)
    LATEST: Tue 12 Nov 16:10:44 UTC 2024 (1731427844)
   RELEASE: default (/lib/firmware/raspberrypi/bootloader/default)
            Use raspi-config to change the release.

Loading the kernel from nvme still fails to get the link up but loading the kernel from USB produces the desired result.