External SSD on raspberry PI 4B SUSPENDED

[blackliner]

I am facing this status every 1-2 days on my setup. I use a raspi as my NAS/backup target (zfs send/receive from PVE) and also to run some docker container (prometheus, grafana and such). I have my docker daemon instructed to use the zpool to store all the files on the SSD. I see no obvious issues here, anyone able to identify something from those dmesg logs?

zpool clear just returns cannot clear errors for backup_pool: I/O error :-(

In this state the disk is also not even visible in lsblk, its like the OS completely put it out of business. (so not sure if I can blame zfs at all 🤷 ). I thought about possible power issues, but the external SSD is the only device attached, and should only draw up to 900mA, and 1.2A should be available.

Setup:

• Raspberry PI 4B 4Gb
• current 64bit raspios release
• Linux raspi01 5.10.92-v8+ # 1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux
• 1Tb external NVMe SSD Transcend Storejet (via USB-C)

zfs --version

zfs-2.0.3-9
zfs-kmod-2.0.3-9

zpool status

  pool: backup_pool
 state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC
  scan: scrub repaired 0B in 01:34:01 with 0 errors on Mon Feb 28 21:31:22 2022
config:

        NAME                                                 STATE     READ WRITE CKSUM
        backup_pool                                          ONLINE       0     0     0
          usb-StoreJet_TS1TESD370C_230152B9G339255F0000-0:0  ONLINE       3    58     0
errors: List of errors unavailable: pool I/O is currently suspended

errors: 19 data errors, use '-v' for a list

dmesg

[ 2483.677674] br-0336396f0e9a: port 1(veth3fb0a57) entered forwarding state
[ 2494.245178] perf: interrupt took too long (3984 > 3983), lowering kernel.perf_event_max_sample_rate to 50000
[ 2518.762463] perf: interrupt took too long (5024 > 4980), lowering kernel.perf_event_max_sample_rate to 39750
[39148.883394] sd 0:0:0:0: [sda] tag#8 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT
[39148.883421] sd 0:0:0:0: [sda] tag#8 CDB: opcode=0x2a 2a 00 12 3a c3 17 00 00 01 00
[39148.892907] sd 0:0:0:0: [sda] tag#5 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD OUT
[39148.892925] sd 0:0:0:0: [sda] tag#5 CDB: opcode=0x2a 2a 00 0a c9 36 6e 00 00 18 00
[39154.771415] sd 0:0:0:0: [sda] tag#7 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD OUT
[39154.771428] sd 0:0:0:0: [sda] tag#7 CDB: opcode=0x2a 2a 00 04 ce ed f1 00 00 08 00
[39154.780993] sd 0:0:0:0: [sda] tag#6 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT
[39154.781002] sd 0:0:0:0: [sda] tag#6 CDB: opcode=0x2a 2a 00 04 ce ed d9 00 00 08 00
[39154.799426] scsi host0: uas_eh_device_reset_handler start
[39154.808366] usb 2-2: cmd cmplt err -2
[39154.817206] usb 2-2: cmd cmplt err -2
[39159.880435] usb 2-2: Disable of device-initiated U1 failed.
[39165.000609] usb 2-2: Disable of device-initiated U2 failed.
[39165.128834] usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[39170.376530] usb 2-2: device descriptor read/8, error -110
[39170.483586] usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[39175.754226] usb 2-2: device descriptor read/8, error -110
[39176.116259] usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[39181.384712] usb 2-2: device descriptor read/8, error -110
[39181.491711] usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[39186.760707] usb 2-2: device descriptor read/8, error -110
[39187.124293] usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[39192.392917] usb 2-2: device descriptor read/8, error -110
[39192.499881] usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[39197.768834] usb 2-2: device descriptor read/8, error -110
[39198.200434] usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[39203.401079] usb 2-2: device descriptor read/8, error -110
[39203.508039] usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[39208.777095] usb 2-2: device descriptor read/8, error -110
[39209.142301] scsi host0: uas_eh_device_reset_handler FAILED err -19
[39209.142334] sd 0:0:0:0: Device offlined - not ready after error recovery
[39209.142349] sd 0:0:0:0: Device offlined - not ready after error recovery
[39209.142362] sd 0:0:0:0: Device offlined - not ready after error recovery
[39209.142374] sd 0:0:0:0: Device offlined - not ready after error recovery
[39209.142427] sd 0:0:0:0: [sda] tag#6 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x06 cmd_age=84s
[39209.142446] sd 0:0:0:0: [sda] tag#6 CDB: opcode=0x2a 2a 00 04 ce ed d9 00 00 08 00
[39209.142467] blk_update_request: I/O error, dev sda, sector 80670169 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[39209.142494] zio pool=backup_pool vdev=/dev/disk/by-id/usb-StoreJet_TS1TESD370C_230152B9G339255F0000-0:0-part1 error=5 type=2 offset=41302077952 size=4096 flags=180880
[39209.142544] sd 0:0:0:0: [sda] tag#7 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x06 cmd_age=84s
[39209.142560] sd 0:0:0:0: [sda] tag#7 CDB: opcode=0x2a 2a 00 04 ce ed f1 00 00 08 00
[39209.142577] blk_update_request: I/O error, dev sda, sector 80670193 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[39209.142596] zio pool=backup_pool vdev=/dev/disk/by-id/usb-StoreJet_TS1TESD370C_230152B9G339255F0000-0:0-part1 error=5 type=2 offset=41302090240 size=4096 flags=180880
[39209.142640] sd 0:0:0:0: [sda] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x06 cmd_age=91s
[39209.142656] sd 0:0:0:0: [sda] tag#5 CDB: opcode=0x2a 2a 00 0a c9 36 6e 00 00 18 00
[39209.142673] blk_update_request: I/O error, dev sda, sector 180958830 op 0x1:(WRITE) flags 0x700 phys_seg 10 prio class 0
[39209.142694] zio pool=backup_pool vdev=/dev/disk/by-id/usb-StoreJet_TS1TESD370C_230152B9G339255F0000-0:0-part1 error=5 type=2 offset=92649872384 size=12288 flags=40080c80
[39209.142712] sd 0:0:0:0: rejecting I/O to offline device
[39209.142737] blk_update_request: I/O error, dev sda, sector 2576 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 0
[39209.142763] zio pool=backup_pool vdev=/dev/disk/by-id/usb-StoreJet_TS1TESD370C_230152B9G339255F0000-0:0-part1 error=5 type=1 offset=270336 size=8192 flags=b08c1
[39209.142783] blk_update_request: I/O error, dev sda, sector 47805035 op 0x1:(WRITE) flags 0x700 phys_seg 4 prio class 0
[39209.142807] zio pool=backup_pool vdev=/dev/disk/by-id/usb-StoreJet_TS1TESD370C_230152B9G339255F0000-0:0-part1 error=5 type=2 offset=24475129344 size=16384 flags=180880
[39209.142841] blk_update_request: I/O error, dev sda, sector 1953506320 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 0
[39209.142865] zio pool=backup_pool vdev=/dev/disk/by-id/usb-StoreJet_TS1TESD370C_230152B9G339255F0000-0:0-part1 error=5 type=1 offset=1000194187264 size=8192 flags=b08c1
[39209.142905] blk_update_request: I/O error, dev sda, sector 142384496 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[39209.142928] blk_update_request: I/O error, dev sda, sector 1953506832 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 0
[39209.142937] zio pool=backup_pool vdev=/dev/disk/by-id/usb-StoreJet_TS1TESD370C_230152B9G339255F0000-0:0-part1 error=5 type=2 offset=72899813376 size=1024 flags=184880
[39209.142958] zio pool=backup_pool vdev=/dev/disk/by-id/usb-StoreJet_TS1TESD370C_230152B9G339255F0000-0:0-part1 error=5 type=1 offset=1000194449408 size=8192 flags=b08c1
[39209.142978] usb 2-2: USB disconnect, device number 2
[39209.143084] blk_update_request: I/O error, dev sda, sector 1371983360 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[39209.143108] zio pool=backup_pool vdev=/dev/disk/by-id/usb-StoreJet_TS1TESD370C_230152B9G339255F0000-0:0-part1 error=5 type=2 offset=702454431744 size=512 flags=180880
[39209.143358] blk_update_request: I/O error, dev sda, sector 1438997604 op 0x1:(WRITE) flags 0x700 phys_seg 4 prio class 0
[39209.143381] zio pool=backup_pool vdev=/dev/disk/by-id/usb-StoreJet_TS1TESD370C_230152B9G339255F0000-0:0-part1 error=5 type=2 offset=736765724672 size=16384 flags=180880
[39209.149338] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[39209.149449] zio pool=backup_pool vdev=/dev/disk/by-id/usb-StoreJet_TS1TESD370C_230152B9G339255F0000-0:0-part1 error=5 type=2 offset=156589501952 size=512 flags=180880
[39209.152072] WARNING: Pool 'backup_pool' has encountered an uncorrectable I/O failure and has been suspended.

[39209.396026] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=0x07 driverbyte=0x00
[39209.544398] usb 2-2: new SuperSpeed Gen 1 USB device number 3 using xhci_hcd
[39214.665024] usb 2-2: device descriptor read/8, error -110
[39214.772070] usb 2-2: new SuperSpeed Gen 1 USB device number 3 using xhci_hcd
[39220.041085] usb 2-2: device descriptor read/8, error -110
[39220.400528] usb 2-2: new SuperSpeed Gen 1 USB device number 4 using xhci_hcd
[39225.673149] usb 2-2: device descriptor read/8, error -110
[39225.780184] usb 2-2: new SuperSpeed Gen 1 USB device number 4 using xhci_hcd
[39231.049182] usb 2-2: device descriptor read/8, error -110
[39231.285213] usb usb2-port2: attempt power cycle
[39231.728628] usb 2-2: new SuperSpeed Gen 1 USB device number 5 using xhci_hcd
[39236.937310] usb 2-2: device descriptor read/8, error -110
[39237.044301] usb 2-2: new SuperSpeed Gen 1 USB device number 5 using xhci_hcd
[39242.313313] usb 2-2: device descriptor read/8, error -110
[39242.744731] usb 2-2: new SuperSpeed Gen 1 USB device number 6 using xhci_hcd
[39247.945380] usb 2-2: device descriptor read/8, error -110
[39248.052430] usb 2-2: new SuperSpeed Gen 1 USB device number 6 using xhci_hcd
[39253.321616] usb 2-2: device descriptor read/8, error -110
[39253.558155] usb usb2-port2: unable to enumerate USB device
[39271.752812] INFO: task txg_sync:592 blocked for more than 120 seconds.
[39271.752837]       Tainted: P         C O      5.10.92-v8+ #1514
[39271.752848] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[39271.752861] task:txg_sync        state:D stack:    0 pid:  592 ppid:     2 flags:0x00000028
[39271.752889] Call trace:
[39271.752918]  __switch_to+0x110/0x168
[39271.752936]  __schedule+0x2d0/0x818
[39271.752949]  schedule+0x48/0x100
[39271.752963]  schedule_timeout+0x190/0x370
[39271.752976]  io_schedule_timeout+0x2c/0x50
[39271.753032]  __cv_timedwait_common+0x110/0x210 [spl]
[39271.753072]  __cv_timedwait_io+0x24/0x30 [spl]
[39271.753361]  zio_wait+0x138/0x2b0 [zfs]
[39271.753620]  dsl_pool_sync+0x3dc/0x470 [zfs]
[39271.753874]  spa_sync+0x528/0xe60 [zfs]
[39271.754127]  txg_sync_thread+0x2c8/0x430 [zfs]
[39271.754169]  thread_generic_wrapper+0x6c/0x84 [spl]
[39271.754192]  kthread+0x148/0x158
[39271.754207]  ret_from_fork+0x10/0x30
[39392.586092] INFO: task txg_sync:592 blocked for more than 241 seconds.
[39392.586113]       Tainted: P         C O      5.10.92-v8+ #1514
[39392.586123] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[39392.586134] task:txg_sync        state:D stack:    0 pid:  592 ppid:     2 flags:0x00000028
[39392.586158] Call trace:
[39392.586184]  __switch_to+0x110/0x168
[39392.586199]  __schedule+0x2d0/0x818
[39392.586210]  schedule+0x48/0x100
[39392.586223]  schedule_timeout+0x190/0x370
[39392.586235]  io_schedule_timeout+0x2c/0x50
[39392.586283]  __cv_timedwait_common+0x110/0x210 [spl]
[39392.586318]  __cv_timedwait_io+0x24/0x30 [spl]
[39392.586566]  zio_wait+0x138/0x2b0 [zfs]
[39392.586791]  dsl_pool_sync+0x3dc/0x470 [zfs]
[39392.587010]  spa_sync+0x528/0xe60 [zfs]
[39392.587228]  txg_sync_thread+0x2c8/0x430 [zfs]
[39392.587264]  thread_generic_wrapper+0x6c/0x84 [spl]
[39392.587281]  kthread+0x148/0x158
[39392.587293]  ret_from_fork+0x10/0x30
[39392.587334] INFO: task dockerd:2252 blocked for more than 120 seconds.
[39392.587344]       Tainted: P         C O      5.10.92-v8+ #1514
[39392.587353] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[39392.587363] task:dockerd         state:D stack:    0 pid: 2252 ppid:     1 flags:0x00000000
[39392.587384] Call trace:
[39392.587398]  __switch_to+0x110/0x168
[39392.587411]  __schedule+0x2d0/0x818
[39392.587422]  schedule+0x48/0x100
[39392.587434]  io_schedule+0x24/0xb0
[39392.587468]  cv_wait_common+0xc4/0x1d0 [spl]
[39392.587501]  __cv_wait_io+0x24/0x30 [spl]
[39392.587726]  txg_wait_synced_impl+0xa8/0x134 [zfs]
[39392.587945]  txg_wait_synced+0x20/0x6c [zfs]
[39392.588204]  dmu_tx_wait+0x258/0x38c [zfs]
[39392.588422]  dmu_tx_assign+0x2d4/0x4d0 [zfs]
[39392.588640]  zfs_write+0x308/0xaa0 [zfs]
[39392.588859]  zpl_iter_write+0xf8/0x180 [zfs]
[39392.588874]  new_sync_write+0x10c/0x190
[39392.588886]  vfs_write+0x254/0x368
[39392.588897]  ksys_write+0x70/0xf8
[39392.588909]  __arm64_sys_write+0x24/0x30
[39392.588924]  el0_svc_common.constprop.3+0xc0/0x1f8
[39392.588936]  do_el0_svc+0x2c/0x98
[39392.588947]  el0_svc+0x20/0x30
[39392.588959]  el0_sync_handler+0x90/0xb8
[39392.588970]  el0_sync+0x180/0x1c0
[39392.588984] INFO: task dockerd:2623 blocked for more than 120 seconds.
[39392.588994]       Tainted: P         C O      5.10.92-v8+ #1514
[39392.589003] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[39392.589013] task:dockerd         state:D stack:    0 pid: 2623 ppid:     1 flags:0x00000008
[39392.589033] Call trace:
[39392.589047]  __switch_to+0x110/0x168
[39392.589058]  __schedule+0x2d0/0x818
[39392.589069]  schedule+0x48/0x100
[39392.589080]  schedule_preempt_disabled+0x20/0x38
[39392.589092]  __mutex_lock.isra.13+0x164/0x520
[39392.589104]  __mutex_lock_slowpath+0x1c/0x28
[39392.589114]  mutex_lock+0x40/0x58
[39392.589333]  zil_commit_impl+0x144/0xe1c [zfs]
[39392.589551]  zil_commit+0x50/0x80 [zfs]
[39392.589769]  zfs_fsync+0x84/0x110 [zfs]
[39392.589987]  zpl_fsync+0x70/0xb0 [zfs]
[39392.590061]  vfs_fsync_range+0x4c/0xa0
[39392.590074]  do_fsync+0x44/0x90
[39392.590086]  __arm64_sys_fsync+0x20/0x30
[39392.590108]  el0_svc_common.constprop.3+0xc0/0x1f8
[39392.590124]  do_el0_svc+0x2c/0x98
[39392.590136]  el0_svc+0x20/0x30
[39392.590151]  el0_sync_handler+0x90/0xb8
[39392.590165]  el0_sync+0x180/0x1c0
[39392.590180] INFO: task dockerd:2624 blocked for more than 120 seconds.
[39392.590204]       Tainted: P         C O      5.10.92-v8+ #1514
[39392.590216] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[39392.590230] task:dockerd         state:D stack:    0 pid: 2624 ppid:     1 flags:0x00000000
[39392.590257] Call trace:
[39392.590274]  __switch_to+0x110/0x168
[39392.590289]  __schedule+0x2d0/0x818
[39392.590303]  schedule+0x48/0x100
[39392.590318]  io_schedule+0x24/0xb0
[39392.590360]  cv_wait_common+0xc4/0x1d0 [spl]
[39392.590397]  __cv_wait_io+0x24/0x30 [spl]
[39392.590625]  txg_wait_synced_impl+0xa8/0x134 [zfs]
[39392.590851]  txg_wait_synced+0x20/0x6c [zfs]
[39392.591075]  zil_commit_impl+0xa04/0xe1c [zfs]
[39392.591313]  zil_commit+0x50/0x80 [zfs]
[39392.591537]  zfs_fsync+0x84/0x110 [zfs]
[39392.591759]  zpl_fsync+0x70/0xb0 [zfs]
[39392.591784]  vfs_fsync_range+0x4c/0xa0
[39392.591800]  do_fsync+0x44/0x90
[39392.591816]  __arm64_sys_fsync+0x20/0x30
[39392.591834]  el0_svc_common.constprop.3+0xc0/0x1f8
[39392.591849]  do_el0_svc+0x2c/0x98
[39392.591864]  el0_svc+0x20/0x30
[39392.591878]  el0_sync_handler+0x90/0xb8
[39392.591892]  el0_sync+0x180/0x1c0
[39392.591997] INFO: task grafana-server:3482 blocked for more than 120 seconds.
[39392.592007]       Tainted: P         C O      5.10.92-v8+ #1514
[39392.592016] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[39392.592026] task:grafana-server  state:D stack:    0 pid: 3482 ppid:  2362 flags:0x00000800
[39392.592046] Call trace:
[39392.592059]  __switch_to+0x110/0x168
[39392.592071]  __schedule+0x2d0/0x818
[39392.592082]  schedule+0x48/0x100
[39392.592093]  io_schedule+0x24/0xb0
[39392.592131]  cv_wait_common+0xc4/0x1d0 [spl]
[39392.592165]  __cv_wait_io+0x24/0x30 [spl]
[39392.592395]  txg_wait_synced_impl+0xa8/0x134 [zfs]
[39392.592623]  txg_wait_synced+0x20/0x6c [zfs]
[39392.592852]  dmu_tx_wait+0x258/0x38c [zfs]
[39392.593072]  dmu_tx_assign+0x2d4/0x4d0 [zfs]
[39392.593301]  zfs_dirty_inode+0x100/0x2a0 [zfs]
[39392.593524]  zpl_dirty_inode+0x34/0x5c [zfs]
[39392.593547]  __mark_inode_dirty+0x1d8/0x4a0
[39392.593562]  generic_update_time+0x88/0xf0
[39392.593574]  inode_update_time+0x34/0x40
[39392.593586]  touch_atime+0x198/0x230
[39392.593810]  zpl_iter_read+0x188/0x1a0 [zfs]
[39392.593829]  new_sync_read+0x10c/0x188
[39392.593847]  vfs_read+0x150/0x1d8
[39392.593859]  ksys_pread64+0x80/0xc8
[39392.593870]  __arm64_sys_pread64+0x28/0x38
[39392.593884]  el0_svc_common.constprop.3+0xc0/0x1f8
[39392.593897]  do_el0_svc+0x2c/0x98
[39392.593912]  el0_svc+0x20/0x30
[39392.593923]  el0_sync_handler+0x90/0xb8
[39392.593935]  el0_sync+0x180/0x1c0
[39392.594062] INFO: task grafana-server:3557 blocked for more than 120 seconds.
[39392.594073]       Tainted: P         C O      5.10.92-v8+ #1514
[39392.594082] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[39392.594095] task:grafana-server  state:D stack:    0 pid: 3557 ppid:  2362 flags:0x00000800
[39392.594117] Call trace:
[39392.594133]  __switch_to+0x110/0x168
[39392.594146]  __schedule+0x2d0/0x818
[39392.594156]  schedule+0x48/0x100
[39392.594167]  io_schedule+0x24/0xb0
[39392.594208]  cv_wait_common+0xc4/0x1d0 [spl]
[39392.594241]  __cv_wait_io+0x24/0x30 [spl]
[39392.594464]  txg_wait_synced_impl+0xa8/0x134 [zfs]
[39392.594693]  txg_wait_synced+0x20/0x6c [zfs]
[39392.594929]  dmu_tx_wait+0x258/0x38c [zfs]
[39392.595156]  dmu_tx_assign+0x2d4/0x4d0 [zfs]
[39392.595394]  zfs_dirty_inode+0x100/0x2a0 [zfs]
[39392.595621]  zpl_dirty_inode+0x34/0x5c [zfs]
[39392.595644]  __mark_inode_dirty+0x1d8/0x4a0
[39392.595657]  generic_update_time+0x88/0xf0
[39392.595668]  inode_update_time+0x34/0x40
[39392.595681]  touch_atime+0x198/0x230
[39392.595911]  zpl_iter_read+0x188/0x1a0 [zfs]
[39392.595926]  new_sync_read+0x10c/0x188
[39392.595937]  vfs_read+0x150/0x1d8
[39392.595953]  ksys_pread64+0x80/0xc8
[39392.595964]  __arm64_sys_pread64+0x28/0x38
[39392.595979]  el0_svc_common.constprop.3+0xc0/0x1f8
[39392.595991]  do_el0_svc+0x2c/0x98
[39392.596003]  el0_svc+0x20/0x30
[39392.596014]  el0_sync_handler+0x90/0xb8
[39392.596025]  el0_sync+0x180/0x1c0
[39392.596039] INFO: task grafana-server:12119 blocked for more than 120 seconds.
[39392.596049]       Tainted: P         C O      5.10.92-v8+ #1514
[39392.596058] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[39392.596068] task:grafana-server  state:D stack:    0 pid:12119 ppid:  2362 flags:0x00000800
[39392.596088] Call trace:
[39392.596102]  __switch_to+0x110/0x168
[39392.596113]  __schedule+0x2d0/0x818
[39392.596129]  schedule+0x48/0x100
[39392.596139]  io_schedule+0x24/0xb0
[39392.596176]  cv_wait_common+0xc4/0x1d0 [spl]
[39392.596209]  __cv_wait_io+0x24/0x30 [spl]
[39392.596436]  txg_wait_synced_impl+0xa8/0x134 [zfs]
[39392.596653]  txg_wait_synced+0x20/0x6c [zfs]
[39392.596873]  dmu_tx_wait+0x258/0x38c [zfs]
[39392.597088]  dmu_tx_assign+0x2d4/0x4d0 [zfs]
[39392.597311]  zfs_dirty_inode+0x100/0x2a0 [zfs]
[39392.597529]  zpl_dirty_inode+0x34/0x5c [zfs]
[39392.597547]  __mark_inode_dirty+0x1d8/0x4a0
[39392.597560]  generic_update_time+0x88/0xf0
[39392.597571]  inode_update_time+0x34/0x40
[39392.597583]  touch_atime+0x198/0x230
[39392.597802]  zpl_iter_read+0x188/0x1a0 [zfs]
[39392.597814]  new_sync_read+0x10c/0x188
[39392.597830]  vfs_read+0x150/0x1d8
[39392.597842]  ksys_pread64+0x80/0xc8
[39392.597853]  __arm64_sys_pread64+0x28/0x38
[39392.597867]  el0_svc_common.constprop.3+0xc0/0x1f8
[39392.597880]  do_el0_svc+0x2c/0x98
[39392.597891]  el0_svc+0x20/0x30
[39392.597901]  el0_sync_handler+0x90/0xb8
[39392.597912]  el0_sync+0x180/0x1c0
[39392.598013] INFO: task node:3740 blocked for more than 120 seconds.
[39392.598028]       Tainted: P         C O      5.10.92-v8+ #1514
[39392.598036] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[39392.598052] task:node            state:D stack:    0 pid: 3740 ppid:  3691 flags:0x00400800
[39392.598080] Call trace:
[39392.598100]  __switch_to+0x110/0x168
[39392.598115]  __schedule+0x2d0/0x818
[39392.598129]  schedule+0x48/0x100
[39392.598144]  io_schedule+0x24/0xb0
[39392.598188]  cv_wait_common+0xc4/0x1d0 [spl]
[39392.598224]  __cv_wait_io+0x24/0x30 [spl]
[39392.598454]  txg_wait_synced_impl+0xa8/0x134 [zfs]
[39392.598682]  txg_wait_synced+0x20/0x6c [zfs]
[39392.598915]  dmu_tx_wait+0x258/0x38c [zfs]
[39392.599137]  dmu_tx_assign+0x2d4/0x4d0 [zfs]
[39392.599370]  zfs_write+0x308/0xaa0 [zfs]
[39392.599592]  zpl_iter_write+0xf8/0x180 [zfs]
[39392.599616]  new_sync_write+0x10c/0x190
[39392.599631]  vfs_write+0x254/0x368
[39392.599646]  ksys_write+0x70/0xf8
[39392.599660]  __arm64_sys_write+0x24/0x30
[39392.599678]  el0_svc_common.constprop.3+0xc0/0x1f8
[39392.599691]  do_el0_svc_compat+0x24/0x50
[39392.599708]  el0_svc_compat+0x20/0x30
[39392.599723]  el0_sync_compat_handler+0xe4/0x110
[39392.599738]  el0_sync_compat+0x184/0x1c0
[39392.599756] INFO: task node:3743 blocked for more than 120 seconds.
[39392.599768]       Tainted: P         C O      5.10.92-v8+ #1514
[39392.599780] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[39392.599793] task:node            state:D stack:    0 pid: 3743 ppid:  3691 flags:0x00400800
[39392.599819] Call trace:
[39392.599836]  __switch_to+0x110/0x168
[39392.599851]  __schedule+0x2d0/0x818
[39392.599865]  schedule+0x48/0x100
[39392.599880]  io_schedule+0x24/0xb0
[39392.599919]  cv_wait_common+0xc4/0x1d0 [spl]
[39392.599956]  __cv_wait_io+0x24/0x30 [spl]
[39392.600189]  txg_wait_synced_impl+0xa8/0x134 [zfs]
[39392.600421]  txg_wait_synced+0x20/0x6c [zfs]
[39392.600650]  dmu_tx_wait+0x258/0x38c [zfs]
[39392.600899]  dmu_tx_assign+0x2d4/0x4d0 [zfs]
[39392.601140]  zfs_dirty_inode+0x100/0x2a0 [zfs]
[39392.601373]  zpl_dirty_inode+0x34/0x5c [zfs]
[39392.601402]  __mark_inode_dirty+0x1d8/0x4a0
[39392.601415]  generic_update_time+0x88/0xf0
[39392.601427]  inode_update_time+0x34/0x40
[39392.601438]  touch_atime+0x198/0x230
[39392.601668]  zpl_iter_read+0x188/0x1a0 [zfs]
[39392.601680]  new_sync_read+0x10c/0x188
[39392.601691]  vfs_read+0x150/0x1d8
[39392.601707]  ksys_read+0x70/0xf8
[39392.601718]  __arm64_sys_read+0x24/0x30
[39392.601733]  el0_svc_common.constprop.3+0xc0/0x1f8
[39392.601745]  do_el0_svc_compat+0x24/0x50
[39392.601757]  el0_svc_compat+0x20/0x30
[39392.601768]  el0_sync_compat_handler+0xe4/0x110
[39392.601780]  el0_sync_compat+0x184/0x1c0
[40240.100673] WARNING: Pool 'backup_pool' has encountered an uncorrectable I/O failure and has been suspended.

[46853.126598] WARNING: Pool 'backup_pool' has encountered an uncorrectable I/O failure and has been suspended.

[46853.130146] WARNING: Pool 'backup_pool' has encountered an uncorrectable I/O failure and has been suspended.

[47153.129438] WARNING: Pool 'backup_pool' has encountered an uncorrectable I/O failure and has been suspended.

[52157.617778] WARNING: Pool 'backup_pool' has encountered an uncorrectable I/O failure and has been suspended.

[84812.653784] WARNING: Pool 'backup_pool' has encountered an uncorrectable I/O failure and has been suspended.

[84812.653868] WARNING: Pool 'backup_pool' has encountered an uncorrectable I/O failure and has been suspended.

[84812.653916] WARNING: Pool 'backup_pool' has encountered an uncorrectable I/O failure and has been suspended.

[84812.653967] WARNING: Pool 'backup_pool' has encountered an uncorrectable I/O failure and has been suspended.

[84812.654019] WARNING: Pool 'backup_pool' has encountered an uncorrectable I/O failure and has been suspended.

[84812.655163] WARNING: Pool 'backup_pool' has encountered an uncorrectable I/O failure and has been suspended.

[rincebrain]

Seems like the drive stops responding then vanishes. I can't really say ZFS is wrong to complain at that point.

You could try forcibly blacklisting UAS, either globally (by blocking loading the uas module) or by making a custom quirks line to block uas with that device ID, in favor of the older storage protocol; just be aware that some drives will only speak UAS, so it's possible this drive will just not show up as a disk after doing this.

[blackliner]

Disabled UAS, will come back with results in a few days. Thanks!

Btw: followed https://forums.raspberrypi.com/viewtopic.php?t=245931

[blackliner]

ok, a week later: looking good, no more write errors, and the disk was never attached. UAS was the issue. Thanks for the help!