ZFS causes random, annyoing disk IO

[segdy]

I use ZFS in my small home server in a single-disk configuration. To save power (and make it less noisy), I spin down the disk after inactivity:

# 120*5=600/60=10min
/sbin/hdparm -S 120 /dev/disk/by-id/ata-ST8000DM004-2CX188_ZCT30YLR

To avoid unnecessary wakups, I zfs-auto-snapshot only takes monthly and weekly snaphots except if the disk is not spun down. For example, I added the following to my /etc/cron.hourly/zfs-auto-snapshot:

smartctl -i -n standby /dev/disk/by-id/ata-ST8000DM004-2CX188_ZCT30YLR >/dev/null; test $? -eq 2 && exit 0

This works mostly and even for extended periods of time. But sometimes, the disk spins up randomly without any purpose whatsoever and without any application requesting disk access.

I activated disk logging and found:

# echo 1 > /proc/sys/vm/block_dump; tail -f /var/log/syslog | grep sda
Jan  5 22:40:17 ed kernel: [1110885.265442@2] z_wr_int(3843090): WRITE block 6208350648 on sda2 (16 sectors)
Jan  5 22:40:17 ed kernel: [1110885.265697@2] z_wr_int(3843090): WRITE block 6208350672 on sda2 (8 sectors)
Jan  5 22:40:17 ed kernel: [1110885.266351@1] z_wr_int(3843091): WRITE block 6208350696 on sda2 (8 sectors)
Jan  5 22:40:17 ed kernel: [1110885.266624@3] z_wr_int(3843091): WRITE block 6208350768 on sda2 (8 sectors)
Jan  5 22:40:17 ed kernel: [1110885.266631@2] z_wr_int(3843093): WRITE block 6208350712 on sda2 (48 sectors)
Jan  5 22:40:17 ed kernel: [1110885.267078@3] z_wr_int(3843092): WRITE block 6208350792 on sda2 (24 sectors)
Jan  5 22:40:17 ed kernel: [1110885.267081@2] z_wr_int(3843093): WRITE block 6987188952 on sda2 (8 sectors)
Jan  5 22:40:17 ed kernel: [1110885.267790@2] z_wr_int(3843092): WRITE block 6987188992 on sda2 (8 sectors)
Jan  5 22:40:17 ed kernel: [1110885.297560@3] z_null_iss(3843073): WRITE block 296 on sda2 (8 sectors)
Jan  5 22:40:17 ed kernel: [1110885.297667@3] z_null_iss(3843073): WRITE block 808 on sda2 (8 sectors)
Jan  5 22:40:17 ed kernel: [1110885.297948@3] z_null_iss(3843073): WRITE block 7812499752 on sda2 (8 sectors)
Jan  5 22:40:17 ed kernel: [1110885.298240@3] z_null_iss(3843073): WRITE block 7812500264 on sda2 (8 sectors)
Jan  5 22:40:22 ed kernel: [1110890.333020@1] z_wr_iss(3843084): WRITE block 7531952400 on sda2 (8 sectors)
Jan  5 22:40:22 ed kernel: [1110890.333078@1] z_wr_iss(3843084): WRITE block 105467504 on sda2 (8 sectors)
Jan  5 22:40:22 ed kernel: [1110890.342857@1] z_wr_int(3843090): WRITE block 105467512 on sda2 (8 sectors)
Jan  5 22:40:22 ed kernel: [1110890.488119@1] z_wr_int(3843091): WRITE block 2832148744 on sda2 (24 sectors)
Jan  5 22:40:22 ed kernel: [1110890.489194@2] z_wr_int(3843091): WRITE block 2987928328 on sda2 (56 sectors)
Jan  5 22:40:22 ed kernel: [1110890.489415@2] z_wr_int(3843091): WRITE block 6982823616 on sda2 (32 sectors)
Jan  5 22:40:22 ed kernel: [1110890.489421@1] z_wr_int(3843094): WRITE block 6141980680 on sda2 (24 sectors)
Jan  5 22:40:22 ed kernel: [1110890.490022@1] z_wr_int(3349993): WRITE block 7531952408 on sda2 (8 sectors)
Jan  5 22:40:22 ed kernel: [1110890.490293@1] z_wr_int(3843093): WRITE block 7721781064 on sda2 (32 sectors)
Jan  5 22:40:22 ed kernel: [1110890.490516@1] z_wr_int(3843092): WRITE block 7757412512 on sda2 (56 sectors)
Jan  5 22:40:22 ed kernel: [1110890.491674@3] z_wr_iss(3843085): WRITE block 7726979352 on sda2 (16 sectors)
Jan  5 22:40:22 ed kernel: [1110890.491711@3] z_wr_iss(3843085): WRITE block 6987189120 on sda2 (16 sectors)
Jan  5 22:40:22 ed kernel: [1110890.491895@1] z_wr_int(3843095): WRITE block 6987189136 on sda2 (8 sectors)
Jan  5 22:40:22 ed kernel: [1110890.492087@2] z_wr_int(3349999): WRITE block 6987189144 on sda2 (8 sectors)
Jan  5 22:40:22 ed kernel: [1110890.492309@2] z_wr_int(3843092): WRITE block 7726979368 on sda2 (16 sectors)
Jan  5 22:40:22 ed kernel: [1110890.492318@3] z_wr_int(3349999): WRITE block 6208350816 on sda2 (32 sectors)
Jan  5 22:40:22 ed kernel: [1110890.492744@1] z_wr_int(3843094): WRITE block 6208350848 on sda2 (40 sectors)

It is the kernel itself that's working on the disk.
There is no process active that should/could cause disk activity.
root and swap is not on ZFS (it is a pure data pool).
I do not think it is the intent log which is written every 5s because a) no user data is written b) most of the time, the disk stays spun down for a very long time.

Are there any maintenance threads that I can disable/tune?

PS: It is not a scheduled zpool scrub either. That happens once a month and is of course OK to spin up the disk.

PPS: Every 30s or so:

# zpool iostat 1
[...]
zpseagate8tb  1.92T  1.70T      0      0      0      0
zpseagate8tb  1.92T  1.70T      0      0      0      0
zpseagate8tb  1.92T  1.70T      0     91      0  1.10M
zpseagate8tb  1.92T  1.70T      0      0      0      0
[...]

[bghira]

looks like it is updating metadata. i think this is an unrealistic request. and you are increasing the LCC of your hard disks by spinning them down.

[segdy]

looks like it is updating metadata.

Yes, this must be the case because I killed all applications which could possibly access data (Samba; it's a data pool, no root or swap) and confirm no files are open on that pool:

# for mp in `zfs list -o mountpoint -H | grep -v '^none'`; do lsof | grep $mp; done
#

Interestingly, meanwhile I exported and re-imported the pool but the activity kept going (for an hour or so).
Then, for the last few hours, it was silent again:

# grep sda /var/log/syslog | tail; date
Jan  6 00:23:04 ed kernel: [1117051.765989@3] z_wr_int(3361637): WRITE block 6677357080 on sda2 (8 sectors)
Jan  6 00:23:04 ed kernel: [1117051.765992@1] z_wr_int(3361630): WRITE block 6677357016 on sda2 (40 sectors)
Jan  6 00:23:04 ed kernel: [1117051.798266@2] z_null_iss(3361615): WRITE block 440 on sda2 (8 sectors)
Jan  6 00:23:04 ed kernel: [1117051.798349@2] z_null_iss(3361615): WRITE block 952 on sda2 (8 sectors)
Jan  6 00:23:04 ed kernel: [1117051.798696@2] z_null_iss(3361615): WRITE block 7812499896 on sda2 (8 sectors)
Jan  6 00:23:04 ed kernel: [1117051.798959@2] z_null_iss(3361615): WRITE block 7812500408 on sda2 (8 sectors)
Jan  6 00:34:35 ed smartd[2600]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 57 to 58
Jan  6 00:34:35 ed smartd[2600]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 43 to 42
Jan  6 01:04:40 ed smartd[2600]: Device: /dev/sda [SAT], is in STANDBY mode, suspending checks
Wed Jan  6 11:33:37 EST 2021
# 

and you are increasing the LCC of your hard disks by spinning them down.

I am aware of that. But this is my private small home server in my living room. I access the data infrequently. Often there is no access for many hours. I am bothered by the unnecessary power consumption and noise level.

I re-emphasize that often spin-down works (see above for example). But then, at random times this z_null_iss kernel thread starts and it doesn't shut up.

[IvanVolosyuk]

'z_wr_int' I think is a worker thread which does actual writing. The initiator of the action is probably something else. We need to go deeper.