Special vdevs and special_small_blocks with zvols

[aaronjwood]

Hi, I am hoping someone can confirm the behavior I see when using special vdevs. I've noticed that the special_small_blocks config doesn't seem to apply for zvols. I don't see any extra data ever going to the special vdevs when writing to my zvol hard. Is this intended? Is there no way to use this feature alongside zvols?

[bghira]

correct, the data portion of the zvol doesn't get used as small blocks. but the metadata does. that can be a substantial benefit.

[rincebrain]

RAW and QCOW2 are format options for storing something representing a disk. You could point e.g. qemu at a file, actual hard drive, or zvol, and tell it to use it as a RAW device, a QCOW2 device, or whatever else, and it would go for it. (I believe anything but RAW is going to require using e.g. qemu-img to format it first, but you get the point.)

If you have a file on a filesystem on ZFS, then if you have special_small_blocks set to nonzero, and the individual records of the file are <= special_small_blocks after compression et al, then ZFS will prefer to store them on the special vdev(s) and fall back to the normal vdevs if there's no space. It doesn't matter what the contents of those files are, what matters is the size of the records in the file.

If the filesystem had a recordsize of 128k and a special_small_blocks of 16k and you made a 40G disk image on it, then unless any of the 128k records compressed to <= 16k, none of the data blocks from that 40G file are gonna end up there.

If you made a zvol with a volblocksize of 16k and set special_small_blocks to 16k, no data blocks from that zvol will ever end up on the special vdev no matter how tiny.

Since the reason for the latter is that there is a hard "if ZVOL block, do not pass go, do not store on special vdev" check, IIRC, it would require a code change to do otherwise.

I hope that makes it clear.

[PrivatePuffin]

Was there any technical reason to have this bypass in the code at all?
I don't see why a zvol block, needs to be excluded from that codepath... because at that point it should both be pretty similair...

[rincebrain]

I'm aware of no technical reason, I suspect the fact that zvol blocks are usually all small and about the same size, so at that point you would more or less be choosing to put the entire zvol on special vdevs, was the reason.

[aaronjwood]

One scenario I can think of for having this (which is what I wanted) is game drives. I have a couple of large zvols where I've set the blocksize to 512K since a huge portion of the data is at or over this size. The rest of the data is much smaller and would have been a perfect candidate for storing on the special vdevs (ssds) since the hdds in the main pool struggle more with all the small files.

[IvanVolosyuk]

Majority of game files are already compressed. It will make quite unlikely that any significant amount of game data will get compressed enough to be stored in the special vdev.
I have a zpool with 128k record size and raw images of game disks + some game files accessible via samba. With 64k special_small_blocks I get only 30G of 800G of the data in the special vdev. Metadata caching is useful though.