ZFS write delays on RPI5 with Ubuntu 23.20-6.5.0-1011-raspi kernel

[git2212]

Hello,

I've compiled and installed zfs 2.2.1 without errors... imported an existing raidz2 array (with x6 disks) again without errors.

While this zpool works fine on other platforms (including a PI 4), there is some strange behavior when writing to the zfs pool. I would describe it as the write freezes every few seconds with no disk IO, then comes to life and burst writes to the disks then hangs again, and so on.
This pool works fine on other devices/platforms with consistent read/writes without freezing like this... so not sure if I need to implement additional configuration steps for the PI 5 platform?

To illustrate the issue I've run the command:

sudo fio --name TEST --eta-newline=5s --filename=/zfs-pool-2/shared/tmp/temp.file --rw=write --size=16g --io_size=10g --blocksize=4096k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=220 --group_reporting

... during the freeze zpool iostat -v reports no disk IO, the output from dstat -tdcD total,mmcblk0,sda,sdb,sdc,sdd,sde,sdf 5 confirms no IO either, see the screenshot...

Reads don't seem to be affected, htop reports no more than a 50% load on the 4 cores at any given time... any ideas are welcome.

Thanks in advance :)

[git2212]

A bit of an update for anyone else having similar issues...

I've spent some time stress testing each drive (part of the pool) separately. I found one of the drives was behaving strange and was stalling the entire pool. Under the stress test the drive was generating log entries such as:

...kernel: scsi host0: uas_eh_device_reset_handler success ...kernel: sd 0:0:0:0: [sda] tag#15 uas_eh_abort_handler 0 uas-tag 6 inflight: CMD IN ...kernel: sd 0:0:0:0: [sda] tag#15 CDB: opcode=0x28 28 00 ad b2 0c 98 00 00 40 00
replaced the drive and the pool performance issues were solved, consistent write throughput is back to the 200Mbps range again ...

... it appears that UAS(P) on the pi 5 is not happy with this one particular disk, why that is is more of a PI 5 issue and not related to OpenZFS in my opinion, as mentioned this issue is absent on a PI 4. There could be 'timing' issues particular to the UAS(P) driver implementation on the pi which don't combine well with a disk which is likely borderline.