ZFS cache speed vs OS cache speed

[rkelleyrtp]

Greetings all. I am trying to find out why ZFS disk cache seems slower than the OS disk cache.

Background: I have a Debian 11 server (VM) running Debian 11 (5.10.0-19-amd64) with 16GB RAM, 8vCPUs, ZFS 2.1.5 (from apt repository). This server has two virtual HDDs - one for EXT4 and the other for ZFS. The ZFS volume has compression=lz4 and a record size of 16K.

Over the past few days, I have noticed files are read much faster from the OS cache than from ZFS cache. I have copied the exact same data to both drives on this server and ran a simple test: Copy the files from the drive to /dev/null. Obviously, I ignore the first reads since the data is not yet in cache. However, subsequent reads come entirely from cache (as verified by htop, dstat, etc).

Test run - EXT-4

run 1 --> root@debian-11:/EXT4/data# time cat 17014* > /dev/null  --> 0m19.131s

run 2 --> root@debian-11:/EXT4/data# time cat 17014* > /dev/null  --> 0m3.728s

run 3 --> root@debian-11:/EXT4/data# time cat 17014* > /dev/null  --> 0m2.614s

run 4 --> root@debian-11:/EXT4/data# time cat 17014* > /dev/null  --> 0m2.605s

run 5 --> root@debian-11:/EXT4/data# time cat 17014* > /dev/null  --> 0m2.600s

Test run - ZFS with lz4 and memory compression enabled

run 1 --> root@debian-11:/ZFS-lz4/data# time cat 17014* > /dev/null  --> 0m34.653s

run 2 --> root@debian-11:/ZFS-lz4/data# time cat 17014* > /dev/null  --> 0m20.762s

run 3 --> root@debian-11:/ZFS-lz4/data# time cat 17014* > /dev/null  --> 0m20.595s

run 4 --> root@debian-11:/ZFS-lz4/data# time cat 17014* > /dev/null  --> 0m20.703s

run 5 --> root@debian-11:/ZFS-lz4/data# time cat 17014* > /dev/null  --> 0m20.639s

Test run - ZFS with lz4 and memory compression DISABLED

run 1 --> root@debian-11:/ZFS-lz4/data# time cat 17014* > /dev/null  --> 0m29.189s

run 2 --> root@debian-11:/ZFS-lz4/data# time cat 17014* > /dev/null  --> 0m10.555s

run 3 --> root@debian-11:/ZFS-lz4/data# time cat 17014* > /dev/null  --> 0m10.378s

run 4 --> root@debian-11:/ZFS-lz4/data# time cat 17014* > /dev/null  --> 0m10.323s

run 5 --> root@debian-11:/ZFS-lz4/data# time cat 17014* > /dev/null  --> 0m10.376s

I am trying to find out why there is such a huge discrepancy between the EXT-4 OS cache speed and ZFS. As you can see from above, the OS cache is much, much faster than ZFS. Are there any tunables that may help to narrow the gap?

[rincebrain]

I'd suggest you ask perf top or perf record or make a FlameGraph to see where your time is being spent, assuming it's not in IOwait - my immediate guess would be that, AIUI, ZFS's cache needs to do a copy to hand someone things from its cache, while Linux, being the thing in charge of the memory subsystem, can just...hand it off.

It taking more time with compressed RAM enabled would track with it needing to make a copy - and thus, compress it. (Unless you meant both LZ4 and memory compression disabled, in which case I would suggest testing with LZ4 and with memory compression disabled.)

[rkelleyrtp]

Thanks for the reply. Seems the culprit is using ZFS RAM compression as well as enabling ADB Scatter. See here for a lengthy discussion. Also, here is another good thread that discusses the performance issue in detail.

TLDR: for max performance when using ZFS compression:

• Disable ARC compression (options zfs zfs_compressed_arc_enabled=0)
• Disable zfs_abd_scatter_enabled (options zfs zfs_abd_scatter_enabled=0)
• Add "init_on_alloc=0 init_on_free=0" to the kernel boot

[rincebrain]

Yeah, I personally think the scatter option is bad and ruins performance in favor of benefits nobody ever actually got around to implementing, but that's just my unscientific $0.02.

I don't see any discussion about why you would want compressed ARC disabled in those threads, but I wouldn't necessarily suggest it as a solution - that's just going to make you always pay the decompression cost upfront, so you're going to be slower on a number of cases, and you lose the substantial memory usage savings by doing it.

There's been discussion of potentially trying to improve performance by optimistically pre-decompressing the next N records when doing sequential IO so that you don't have to wait on that if you can readily predict what's going to be needed next, but I don't know if/when someone's going to try implementing that...

[rkelleyrtp]

For us, the trade off is performance vs amount of data in ARC. In the end, trying to keep most of the data in RAM (even compressed) was not practical. Thus, we opted for speed over ARC capacity.

[gdevenyi]

Another case for updated performance documentation, #13390

[rincebrain]

Sure, I'd argue that in most cases if you need to turn that off, it's probably something that needs to be improved, rather than your use case requiring it, which was why I mentioned the potential for something that optimistically decompresses so you can avoid it being the blocking factor for your use case.

Not saying you shouldn't do it if it works for you, obviously, just that I would say if we add a note about doing that to documentation, for example, it should probably include "if you find you need to do this for your use case, please let us know, we'd like to improve it".