Ubuntu Server 20.04.3 LTS - ZFS import by serial

[douglim]

Hardware: Raspberry Pi 4 Model B Rev 1.4 - 8 GB RAM
OS: Ubuntu Server 20.04.3 LTS
Kernel: Linux fs 5.4.0-1045-raspi #49-Ubuntu SMP PREEMPT Wed Sep 29 17:49:16 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

The currently installed zfsutils-linux version is 0.8.3

I'm setting up ZFS on a fresh install of Ubuntu Server for Raspberry Pi. It's a currently just a test setup. There's no production data at risk, so I can completely wipe and restart with a fresh install as needed.

I'm setting up an 8 x 6 TB raidz2 pool using 6 TB SATA drives in (2) four-bay SATA/USB3 docks.

I have no problem creating the pool using /dev/sda .. /dev/sdh

# zpool create pool1 raidz2 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh
# zpool status pool1
  pool: pool1
 state: ONLINE
  scan: resilvered 36K in 0 days 00:00:00 with 0 errors on Wed Nov 10 17:02:44 2021
config:

        NAME        STATE     READ WRITE CKSUM
        pool1       ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            sda     ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0
            sde     ONLINE       0     0     0
            sdf     ONLINE       0     0     0
            sdg     ONLINE       0     0     0
            sdh     ONLINE       0     0     0

Once created, I am able to read/write some test data - disk benchmarking with sysbench, for example.

The physical drives don't get consistently assigned to /dev/sd* device files across reboots, so I want to have disks assigned to the pool by serial number, so I have a custom udev rule:

/lib/udev/rules.d/10-local.rules
KERNEL=="sd[a-h]", SUBSYSTEM=="block", PROGRAM="get_disk_serial.sh %k", SYMLINK+="disk/by-serial/%c"

At boot /dev/disk/by-serial looks like:

# ls -l /dev/disk/by-serial/
total 0
lrwxrwxrwx 1 root root 9 Nov 10 17:02 WD-C80J0VVG -> ../../sdg
lrwxrwxrwx 1 root root 9 Nov 10 17:02 WD-C80JB4KG -> ../../sda
lrwxrwxrwx 1 root root 9 Nov 10 17:02 WD-C80JEAHG -> ../../sdh
lrwxrwxrwx 1 root root 9 Nov 10 17:02 WD-C80JLDTG -> ../../sdf
lrwxrwxrwx 1 root root 9 Nov 10 17:02 WD-C80JLDXG -> ../../sdb
lrwxrwxrwx 1 root root 9 Nov 10 17:02 WD-C80JT3WG -> ../../sdd
lrwxrwxrwx 1 root root 9 Nov 10 17:02 WD-C80JT4SG -> ../../sdc
lrwxrwxrwx 1 root root 9 Nov 10 17:02 WD-C80JWWGG -> ../../sde

I should be able to export pool1 and then reimport using the links in /dev/disk/by-serial. I get the following:

# zpool export pool1
# zpool import -d /dev/disk/by-serial pool1
cannot import 'pool1': no such pool available

Any ideas on how to troubleshoot/solve this. At first glance, this looks like it might be a problem with the zpool command handling the -d option, rather than some issue with how the physical disks are linked in /dev/disk/by-serial. I could be wrong.

[gdevenyi]

/dev/disk/by-id/ is already a thing.

[douglim]

Thanks. I'm aware of /dev/disk/by-id

Unfortunately, the ids assigned correspond to the enclosure and slot of the SATA docks, not anything corresponding to the physical disks, which is less than ideal. Shouldn't the -d option be expected to work with any directory of links to disks in /dev?

[gdevenyi]

Very odd. My /dev/disk/by-id contains:
ata-ST3500418AS_<serial>

Your use of -d should work if your symlinks are correct.

[douglim]

Are your disks connected to the host directly via SATA or through a USB/SATA adapter? I would expect /dev/disk/by-serial to work as expected (as on your system) if it were a direct host/SATA connection.

The symlinks to /dev/sd* are identical, except /dev/disk/by-serial does not include partition symlinks that /dev/disk/by-id does. I don't believe that should matter, but I could be wrong.

# ls -l /dev/disk/by-id/
total 0

lrwxrwxrwx 1 root root 13 Nov 11 10:06 mmc-SD32G_0x7ce66f71 -> ../../mmcblk0
lrwxrwxrwx 1 root root 15 Nov 11 10:06 mmc-SD32G_0x7ce66f71-part1 -> ../../mmcblk0p1
lrwxrwxrwx 1 root root 15 Nov 11 10:06 mmc-SD32G_0x7ce66f71-part2 -> ../../mmcblk0p2
lrwxrwxrwx 1 root root  9 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004ABA-0:0 -> ../../sde
lrwxrwxrwx 1 root root 10 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004ABA-0:0-part1 -> ../../sde1
lrwxrwxrwx 1 root root 10 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004ABA-0:0-part9 -> ../../sde9
lrwxrwxrwx 1 root root  9 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004ABA-0:1 -> ../../sdf
lrwxrwxrwx 1 root root 10 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004ABA-0:1-part1 -> ../../sdf1
lrwxrwxrwx 1 root root 10 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004ABA-0:1-part9 -> ../../sdf9
lrwxrwxrwx 1 root root  9 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004ABA-0:2 -> ../../sdg
lrwxrwxrwx 1 root root 10 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004ABA-0:2-part1 -> ../../sdg1
lrwxrwxrwx 1 root root 10 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004ABA-0:2-part9 -> ../../sdg9
lrwxrwxrwx 1 root root  9 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004ABA-0:3 -> ../../sdh
lrwxrwxrwx 1 root root 10 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004ABA-0:3-part1 -> ../../sdh1
lrwxrwxrwx 1 root root 10 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004ABA-0:3-part9 -> ../../sdh9
lrwxrwxrwx 1 root root  9 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004B24-0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004B24-0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004B24-0:0-part9 -> ../../sda9
lrwxrwxrwx 1 root root  9 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004B24-0:1 -> ../../sdb
lrwxrwxrwx 1 root root 10 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004B24-0:1-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004B24-0:1-part9 -> ../../sdb9
lrwxrwxrwx 1 root root  9 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004B24-0:2 -> ../../sdc
lrwxrwxrwx 1 root root 10 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004B24-0:2-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004B24-0:2-part9 -> ../../sdc9
lrwxrwxrwx 1 root root  9 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004B24-0:3 -> ../../sdd
lrwxrwxrwx 1 root root 10 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004B24-0:3-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Nov 11 10:06 usb-Sabrent_Disk_Device_000000004B24-0:3-part9 -> ../../sdd9

# ls -l /dev/disk/by-serial/
total 0

lrwxrwxrwx 1 root root 9 Nov 11 10:06 WD-C80J0VVG -> ../../sdg
lrwxrwxrwx 1 root root 9 Nov 11 10:06 WD-C80JB4KG -> ../../sda
lrwxrwxrwx 1 root root 9 Nov 11 10:06 WD-C80JEAHG -> ../../sdh
lrwxrwxrwx 1 root root 9 Nov 11 10:06 WD-C80JLDTG -> ../../sdf
lrwxrwxrwx 1 root root 9 Nov 11 10:06 WD-C80JLDXG -> ../../sdb
lrwxrwxrwx 1 root root 9 Nov 11 10:06 WD-C80JT3WG -> ../../sdd
lrwxrwxrwx 1 root root 9 Nov 11 10:06 WD-C80JT4SG -> ../../sdc
lrwxrwxrwx 1 root root 9 Nov 11 10:06 WD-C80JWWGG -> ../../sde

It's not obvious, but I did not create explicitly partition 1 and 9 on each disk. The partitions get created during the ZFS pool creation after starting from a completely wiped disk with wipefs -af </dev block file>

zpool import -d /dev/disk/by-id pool1 is workable. It's not a major burden to make sure that the physical drives stay in the same physical dock slots. It's workable, but I'm not seeing any reason why using the by-serial path shouldn't work.

[gdevenyi]

My disks are either SAS or SATA connected (via direct connection, or SAS backplanes). I don't have any partitions for zfs-created pools.

Its clear your USB enclosure is doing some.. questionable things, not passing the ATA commands properly to give the real disk IDs. As far as I understood, zfs doesn't need the devices to be the same name on each import, as it scans for the magic signatures on the disks. As such, the by-id showing the slots is probably the most useful.