Reducing "deadman" warnings when sending snapshots to an s3backer-backed pool

[alaric728]

System information

Debian | Buster
4.19.208 | amd64
OpenZFS | 2.0.8-12

Describe the problem you're observing

Howdy!

I've got a bit of an odd one for you!

I'm trying to send snapshots from a pool of local disks to a pool supported by s3backer. I'm doing something like:

zfs send -L -c -i local_pool/fs@snap1 local_pool/fs@snap2 | zfs receive -s -F -d s3pool

Where I've got the s3pool mounted locally with these parameters (access key, etc. redacted):
--size=10T,--blockSize=1M,--listBlocks,--ssl,--debug,--debug-http,--directIO

I'm experiencing pretty massive iowait within ZFS when sending bulk data through the bucket. The write speed appears to be about 100 MB/s actual, but according tools like pv the stream between the two programs is about 1GB/s. When I write more than 30 GB of data (the amount that I can write in <6 minutes, which will become apparent shortly), zed starts blowing up the logs with deadman timeouts:

Feb 16 17:47:06 host zed[26466]: eid=185323 class=deadman pool='s3pool' vdev=file size=1048576 offset=929096007680 priority=3 err=0 flags=0x184880 bookmark=75:11:0:86276
Feb 16 17:47:06 host zed[26477]: eid=185324 class=deadman pool='s3pool' vdev=file size=1048576 offset=929094959104 priority=3 err=0 flags=0x184880 bookmark=75:11:0:86275
Feb 16 17:47:06 host zed[26489]: eid=185325 class=deadman pool='s3pool' vdev=file size=1048576 offset=929093910528 priority=3 err=0 flags=0x184880 bookmark=75:11:0:86274
Feb 16 17:47:06 host zed[26497]: eid=185326 class=deadman pool='s3pool' vdev=file size=1048576 offset=929092861952 priority=3 err=0 flags=0x184880 bookmark=75:11:0:86273
Feb 16 17:47:06 host zed[26510]: eid=185327 class=deadman pool='s3pool' vdev=file size=1048576 offset=929091813376 priority=3 err=0 flags=0x184880 bookmark=75:11:0:86272
Feb 16 17:47:06 host zed[26521]: eid=185328 class=deadman pool='s3pool' vdev=file size=1048576 offset=929090764800 priority=3 err=0 flags=0x184880 bookmark=75:11:0:86271
Feb 16 17:47:06 host zed[26531]: eid=185329 class=deadman pool='s3pool' vdev=file size=1048576 offset=929089716224 priority=3 err=0 flags=0x184880 bookmark=75:11:0:86270
Feb 16 17:47:06 host zed[26541]: eid=185330 class=deadman pool='s3pool' vdev=file size=1048576 offset=929088667648 priority=3 err=0 flags=0x184880 bookmark=75:11:0:86269
Feb 16 17:47:06 host zed[26556]: eid=185331 class=deadman pool='s3pool' vdev=file size=1048576 offset=929087619072 priority=3 err=0 flags=0x184880 bookmark=75:11:0:86268
Feb 16 17:47:06 host zed[26566]: eid=185332 class=deadman pool='s3pool' vdev=file size=1048576 offset=929086570496 priority=3 err=0 flags=0x184880 bookmark=75:11:0:86267
Feb 16 17:47:06 host zed[26576]: eid=185333 class=deadman pool='s3pool' vdev=file size=1048576 offset=929085521920 priority=3 err=0 flags=0x184880 bookmark=75:11:0:86266
Feb 16 17:47:06 host zed[26591]: eid=185334 class=deadman pool='s3pool' vdev=file size=1048576 offset=929084473344 priority=3 err=0 flags=0x184880 bookmark=75:11:0:86265
Feb 16 17:47:06 host zed[26604]: eid=185335 class=deadman pool='s3pool' vdev=file size=1048576 offset=929083424768 priority=3 err=0 flags=0x184880 bookmark=75:11:0:86264 

Furthermore, when the writes actually complete, I get delay numbers that seem impossiby long:

5,3,2022-02-16T18:56:01.443117-08:00,host,zed: eid=221508 class=deadman pool='s3pool' vdev=file size=1048576 offset=1346134605824 priority=3 err=0 flags=0x184880 delay=11384119ms bookmark=556:11:0:71361

Here the delay reports as roughly 3 hours, when the send to about 20 minutes to complete.

I sort of expect that making a zpool out of a single file that's supported by s3backer is a tad far out of the zfs use-case, but it would be great if I could figure out a way to configure it so I didn't blow up my logs everytime I try to send a snapshot back up for DR.

Does anyone have a good solution for this? I looked at tuning deadman times higher, but that wouldn't help much since I'd eventually outpace it. Is there some kind of behavior I can turn on to get ZFS to back-off on write speed on this particular pool?