Tuning to minimize fragmentation?

[richardm1]

I recall reading (years ago) a discussion surrounding ZFS tunings to minimize fragmentation -- possibly at the expense of data security and overall performance. I think it went along the lines of highly delayed writes to maximize coalescence.

Am I imagining this?

[amotin]

Fragmentation is produced by specific application workloads. One of prominent examples is torrent, downloading blocks in random order and so creating fragmentation, since ZFS on a large scale writes the data in the order of arrival (rounded by transaction groups), not the logical order within the files. In case of at least Transmission torrent client it is possible to set up separate small pool for incoming files, and then it will move the completed ones to the main pool sequentially without fragmentation.

Theoretically by blowing transaction group size in ZFS hard you may reduce some of fragmentation, since within each TXG data are written in logical order, but you can do that only up to certain percent of your ARC size. The default is 10% or 4GB, so there can be some space for possible tuning, but not much. In case of torrents fragmentation is produced by hours and days of workload, not some seconds. You can't stretch transaction group to hours.

[richardm1]

Thanks. I've been playing around with this today. I have the disks sitting idle for a good 15 seconds or so and then a burst of write activity that saturates all four spindles, then back to idle, repeat. I started with zfs_txg_timeout=30 then I cranked up these guys:

zfs_vdev_async_write_active_min_dirty_percent
zfs_vdev_async_write_active_max_dirty_percent
zfs_dirty_data_max_percent
zfs_dirty_data_sync_percent
zfs_delay_min_dirty_percent

Years ago I had a sweet set of dtrace scripts for BSD that would clearly show txg events/sizes, ZIL activity, and so forth. However it seems like dtrace for Linux is kind of an afterthought and doesn't enjoy much popularity or support. How are people getting relevant telemetry from Linux beyond zpool iostat and arcstat.py?

[filip-paczynski]

Hi, I think having a SLOG also helps. Not only with fragmentation of data, but also something related to metaslabs... sorry I don't remember exactly - there was a video from ZFS conference about it.

[richardm1]

Hi, I think having a SLOG also helps.

I remember reading that as well. Something about once a sync write has been committed to the SLOG device it's no longer urgent therefore it gets coalesced with the async writes and written on their lazier schedule. I'm using a 3.8G slice of an Optane drive for this.

I'd love to have a script that shows what's going on with SLOG in real-time. Again, I had something for this when I was playing with FreeNAS in 2018 but I've nothing for Linux. zpool iostat shows data being written to the device but I can't see how much uncommitted data is sitting there at any point in time.

[rincebrain]

A specific reason that comes to mind that it can help is that a slog effectively can significantly reduce how much you fragment your free space by doing the transient allocations out of a dedicated pool of space rather than putting short-lived allocations in random places. A change went in to pick one specific metaslab to always do ZIL allocations out of, to minimize the impact of them on free space fragmentation elsewhere...

cf. aa755b3 and #11389 which are in 2.1 and newer.

As far as live stats, dynamic probing on Linux aside, you might find useful data in /proc/spl/kstat/zfs/... or /proc/spl/kstat/zfs/[poolname]/...