Why do small files still get written to the main pool VDEVs when a special Allocation Class VDEV is active and being written to?

[recklessnl]

System information

Type	Proxmox VE 6
Distribution Name	Debian
Distribution Version	10
Linux Kernel	5.0.23
Architecture	amd64
ZFS Version	0.8.1
SPL Version	0.8.1-pve2

Describe the problem you're observing

I'm running the following fio tests to benchmark special allocation classes:

fio --filename=test_fio --sync=1 --rw=write --bs=32K --numjobs=4 --iodepth=32 --runtime=60 --time_based --group_reporting --name=test --size=32K

My main pool (called pool) consists of 9 striped mirrorred harddisks (18 disks in total) and 2 striped mirrored SSDs (Intel Datacenter SSDs) as a special VDEV (4 in total). The random writes and reads of these SSDs, compared to the rust hard disks, as we all know, is night and day.

I have 2 datasets: pool/with and pool/without.
pool/with has special_small_blocks set to 128K. pool/without does not have that setting enabled at all.

Whenever I run the above fio benchmark, I do not see any difference in performance at all. In fact, I tend to see worse performance. What I expect is a big difference, as random writes will go to the SSDs in the special VDEV in pool/with, and in pool/without` they will go to the hard drives.

When I watch the output of zpool iostat pool -v 1 , I notice that when I do the above fio test to pool/without, the special VDEV does not get any writes to it. This is to be expected.
However, when I run the same command at pool/with, I notice that the main hard drives still get writes to them. How is that possible?

Shouldn't the SSD take over the complete whole workload in this scenario? fio creates four 32K files, and they should go to the special VDEVs. Is this expected behavior and if so, what is the point of the special_small_blocks? It seems to not help with this benchmark at all.

And why does the performance not increase with having special_small_blocks set?

[recklessnl]

Hi @behlendorf, do you have any idea what this could be?

[scineram]

Isn’t that just a sync write test? There is no log device.

[thulle]

It would makes sense that the special vdev would be preferred as ZIL over the other (except slog), since it's likely to be the quicker one. If that preference is not the case then the sync writes would behave like this.

@recklessnl what happens if you run it without sync?

[recklessnl]

@thulle The speeds are amazing without sync, due to it writing it to RAM first.

If there is a better way to properly benchmark the situation laid out in the OP, please let me know.

[thulle]

@recklessnl That's not in question, the question is whether the writes go to the special vdev or the main pool.

Sync writes go both to ZIL and special vdev. If the ZIL is on the main pool you will write to both main pool and special vdev.
Async writes doesn't have to be writte to the ZIL, so writes should go straight to the special vdev.

If async writes are seen on the main pool, there might be some bug here in selecting small blocks for the special vdev.
If you add a SLOG the sync writes should end up on SLOG and Special vdev instead of pool and special vdev.

edit: I'm not using correct terminology, main pool = harddrive VDEV.