Research booting NanoPi R6C

[jakubgs]

Research into running NixOS on a NanoPi R6C device.

Links:

• https://www.friendlyelec.com/index.php?route=product/product&product_id=291
• https://wiki.friendlyelec.com/wiki/index.php/NanoPi_R6C

[jakubgs]

I installed Ubuntu using rk3588-eflasher-ubuntu-jammy-minimal-5.10-arm64-20230527.img image. Here's the storage:

pi@NanoPi-R6C:~$ sudo fdisk -l /dev/mmcblk2 /dev/nvme0n1 
Disk /dev/mmcblk2: 28.91 GiB, 31037849600 bytes, 60620800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 73987B6B-4974-4C94-A3E8-58AB2EB7A946

Device           Start      End  Sectors  Size Type
/dev/mmcblk2p1   16384    24575     8192    4M unknown
/dev/mmcblk2p2   24576    32767     8192    4M unknown
/dev/mmcblk2p3   32768    40959     8192    4M unknown
/dev/mmcblk2p4   40960    73727    32768   16M unknown
/dev/mmcblk2p5   73728   155647    81920   40M unknown
/dev/mmcblk2p6  155648   221183    65536   32M unknown
/dev/mmcblk2p7  221184   286719    65536   32M unknown
/dev/mmcblk2p8  286720  5267455  4980736  2.4G unknown
/dev/mmcblk2p9 5267456 60620766 55353311 26.4G unknown


Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors
Disk model: SAMSUNG MZVLB512HAJQ-000L7              
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

[jakubgs]

aWe can get more insight into the partitions on the eMMC by using parted:

pi@NanoPi-R6C:~$ sudo parted /dev/mmcblk2 print
Model: MMC A3A551 (sd/mmc)
Disk /dev/mmcblk2: 31.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name      Flags
 1      8389kB  12.6MB  4194kB               uboot
 2      12.6MB  16.8MB  4194kB               misc
 3      16.8MB  21.0MB  4194kB               dtbo
 4      21.0MB  37.7MB  16.8MB               resource
 5      37.7MB  79.7MB  41.9MB               kernel
 6      79.7MB  113MB   33.6MB               boot
 7      113MB   147MB   33.6MB               recovery
 8      147MB   2697MB  2550MB  ext4         rootfs
 9      2697MB  31.0GB  28.3GB  ext4         userdata

https://wiki.friendlyelec.com/wiki/index.php/NanoPi_R6C#Install_to_target_board

[jakubgs]

Apparently the kernel needs to be written to the 5th partition based on instructions:

dd if=resource.img of=/dev/mmcblk2p4 bs=1M
dd if=kernel.img of=/dev/mmcblk2p5 bs=1M
dd if=uboot.img of=/dev/mmcblk2p1 bs=1M

And the Ubuntu image actually has no kernel or configuration in /boot:

pi@NanoPi-R6C:~$ ls -l /boot
total 0

Which is different from how NanoPC-T4 works, since there you actually have an extlinux config file in /boot and kernels.

[jakubgs]

I put in it a Samsung PM981 512 GB M.2 2280 (MZVLB512HAJQ) NVMe and here's some very basic benchmarks:

 > sudo hdparm -tT /dev/nvme0n1
/dev/nvme0n1:
 Timing cached reads:   8034 MB in  2.00 seconds = 4020.01 MB/sec
 Timing buffered disk reads: 1030 MB in  3.00 seconds = 343.10 MB/sec

/dev/nvme0n1:
 Timing cached reads:   8342 MB in  2.00 seconds = 4174.52 MB/sec
 Timing buffered disk reads: 1032 MB in  3.00 seconds = 343.92 MB/sec

/dev/nvme0n1:
 Timing cached reads:   8054 MB in  2.00 seconds = 4029.94 MB/sec
 Timing buffered disk reads: 1032 MB in  3.00 seconds = 343.72 MB/sec

 > sudo dd if=/dev/zero of=/mnt/test.img bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.06961 s, 528 MB/s

2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.77926 s, 449 MB/s

2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.48503 s, 479 MB/s

[jakubgs]

Some potentially interesting resources:

• https://gitlab.com/leonardp/nanopi-r4s-nixos - Someone's U-Boot and SD card image for Nanopi R4S.
• https://github.com/samueldr/Tow-Boot/tree/wip/rock5-vendor - Samueldr setup for building U-Boot for Radxa ROCK 5B
• https://wiki.radxa.com/Rock5/guide/build-u-boot-on-5b - Instructions for building U-Boot for ROCK B5
• https://u-boot.readthedocs.io/en/latest/board/amlogic/nanopi-k2.html - Instructions for building U-Boot for NanoPi K2
• https://wiki.friendlyelec.com/wiki/index.php/Building_U-boot_and_Linux_for_H5/H3/H2%2B - U-Boot for NanoPi H2/H3/H5

[jakubgs]

I have tried booting a generic SD card image of Nixos, since it works on NanoPC-T4, but it does not work:
https://hydra.nixos.org/job/nixos/trunk-combined/nixos.sd_image.aarch64-linux

[jakubgs]

I built the image:

 > ls -l $(nix-build -A radxa-rock5b.config.build.firmwareSPI)/binaries
trace: Using default Nixpkgs revision '1905f5f2e55e0db0bb6244cfe62cb6c0dbda391d'...
trace: ***********************************
trace: * Evaluating device: radxa-rock5b *
trace: ***********************************
trace: Building with crossSystem?: aarch64-linux != x86_64-linux → we are.
trace:     crossSystem: config: aarch64-unknown-linux-gnu
trace: Building with crossSystem?: aarch64-linux != x86_64-linux → we are.
trace:     crossSystem: config: aarch64-unknown-linux-gnu
total 1796
-r--r--r-- 1 root root 1866752 Jan  1  1970 Tow-Boot.spi.bi

But it's not clear how I'm supposed to install it. But these instructions for ROCK 5B just use dd:

sudo dd if=spi-image.img of=/dev/mtdblock0

https://wiki.radxa.com/Rock5/install/spi

[jakubgs]

I tried installing it in the U-Boot partition to see if I can maybe get the generic SD image to boot with it:

pi@NanoPi-R6C:~$ sudo dd if=/dev/zero of=/dev/mmcblk2p1
dd: writing to '/dev/mmcblk2p1': No space left on device
8193+0 records in
8192+0 records out
4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.410807 s, 10.2 MB/s

pi@NanoPi-R6C:~$ sudo dd if=Tow-Boot.spi.bin of=/dev/mmcblk2p1
3646+0 records in
3646+0 records out
1866752 bytes (1.9 MB, 1.8 MiB) copied, 0.153187 s, 12.2 MB/s

But of course that did not work. And now nothing boots. That was pure wishful thinking.

[jakubgs]

Using picocom with baud 1500000 via a USB debug port I was able to get the UART to work:

 > sudo minicom -b 1500000 -D /dev/ttyUSB0
...
DDR V1.10 75d050770f typ 23/02/28-20:47:10
LPDDR4X, 2112MHz
channel[0] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB
channel[1] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB
channel[2] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB
channel[3] BW=16 Col=10 Bk=8 CS0 Row=16 CS1 Row=16 CS=2 Die BW=16 Size=2048MB
Manufacturer ID:0x1
CH0 RX Vref:26.3%, TX Vref:21.8%,21.8%
CH1 RX Vref:26.3%, TX Vref:22.8%,21.8%
CH2 RX Vref:27.5%, TX Vref:19.8%,18.8%
CH3 RX Vref:26.7%, TX Vref:20.8%,21.8%
change to F1: 528MHz
change to F2: 1068MHz
change to F3: 1560MHz
change to F0: 2112MHz
out
U-Boot SPL board init
U-Boot SPL 2017.09-gd91b448214-221107 #root (Apr 10 2023 - 14:05:21)
unknown raw ID 0 0 0
unrecognized JEDEC id bytes: 00, 00, 00
Trying to boot from MMC2
part_get_info_efi: *** ERROR: Invalid GPT ***
part_get_info_efi: ***        Using Backup GPT ***
Trying fit image at 0x4000 sector
Not fit magic
Trying fit image at 0x5000 sector
Not fit magic
Trying fit image at 0x4000 sector
Not fit magic
Trying fit image at 0x5000 sector
Not fit magic
Trying to boot from MMC1
Trying fit image at 0x4000 sector
Not fit magic
Trying fit image at 0x5000 sector
Not fit magic
Trying fit image at 0x4000 sector
Not fit magic
Trying fit image at 0x5000 sector
Not fit magic
SPL: failed to boot from all boot devices
### ERROR ### Please RESET the board ###
# Reset the board to bootrom #

As we can see it fails to boot: SPL: failed to boot from all boot devices
I wonder how this looked when I had a working Ubuntu image. Lets see.

[jakubgs]

There are instructions on how to build U-Boot from FriendlyELEC source:

git clone https://github.com/friendlyarm/uboot-rockchip --depth 1 -b nanopi6-v2017.09
UBOOT_SRC=$PWD/uboot-rockchip ./build-uboot.sh friendlycore-focal-arm64

https://wiki.friendlyelec.com/wiki/index.php/NanoPi_R6C#Compile_the_uboot

[jakubgs]

The Ubuntu flash image boots without issues:

U-Boot SPL board init
U-Boot SPL 2017.09-gd91b448214-221107 #root (Apr 10 2023 - 14:05:21)
unknown raw ID 0 0 0
unrecognized JEDEC id bytes: 00, 00, 00
Trying to boot from MMC2
Trying fit image at 0x4000 sector
## Verified-boot: 0
## Checking atf-1 0x00040000 ... sha256(ac799e4aa0...) + OK
## Checking uboot 0x00200000 ... sha256(ecc2ca052f...) + OK
## Checking fdt 0x0032ce40 ... sha256(e936f08b25...) + OK
## Checking atf-2 0xff100000 ... sha256(c0f2f7769f...) + OK
## Checking atf-3 0x000f0000 ... sha256(338cd7cb83...) + OK
## Checking optee 0x08400000 ... sha256(fde0860845...) + OK
Jumping to U-Boot(0x00200000) via ARM Trusted Firmware(0x00040000)

boot.log

[jakubgs]

At least we can see the NVMe shows up properly at boot here, so maybe NixOS won't have same issue as on NanoPC-T4.

[    4.475674] nvme nvme0: pci function 0004:41:00.0
[    4.475754] nvme 0004:41:00.0: enabling device (0000 -> 0002)
[    4.478778] nvme nvme0: Shutdown timeout set to 8 seconds
[    4.497511] nvme nvme0: 8/0/0 default/read/poll queues
[    4.524958] rk-pcie fe180000.pcie: PCIe Link up, LTSSM is 0x130011
[    4.525205] rk-pcie fe180000.pcie: PCI host bridge to bus 0003:30

boot.log

[jakubgs]

If we take a look at an SD card image for Ubuntu we can see the same layout as on the eMMC:

rk3588-eflasher-ubuntu-jammy-minimal-5.10-arm64-20230527.img

 > sudo fdisk -l /dev/loop3 
Disk /dev/loop3: 7.26 GiB, 7799999488 bytes, 15234374 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 73987B6B-4974-4C94-A3E8-58AB2EB7A946

Device         Start      End  Sectors  Size Type
/dev/loop3p1 3563520 15228927 11665408  5.6G unknown
/dev/loop3p2   16384    24575     8192    4M unknown
/dev/loop3p3   24576    32767     8192    4M unknown
/dev/loop3p4   32768    40959     8192    4M unknown
/dev/loop3p5   40960    73727    32768   16M unknown
/dev/loop3p6   73728   155647    81920   40M unknown
/dev/loop3p7  155648   221183    65536   32M unknown
/dev/loop3p8  221184   286719    65536   32M unknown
/dev/loop3p9  286720  3563519  3276800  1.6G unknown

Partition table entries are not in disk order.

While the NixOS image uses a FAT32 partition for the boot files, which I guess is why it doesn't work:

 > sudo fdisk -l /dev/loop2 
Disk /dev/loop2: 2.54 GiB, 2723512320 bytes, 5319360 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x2178694e

Device       Boot Start     End Sectors  Size Id Type
/dev/loop2p1      16384   77823   61440   30M  b W95 FAT32
/dev/loop2p2 *    77824 5319359 5241536  2.5G 83 Linux

[jakubgs]

Here are official RockChip docs about the boot process: https://opensource.rock-chips.com/wiki_Boot_option