Another question about special vdev small block feature

[Hr46ph]

I read this: #14542
And a whole lot of other info, some of it contradictory and Im not sure what to believe.

My confusion lies in blocks, records and files. The special vdev is said to accept "small blocks" (not files), and explanations point often to the a dataset's recordsize to explain what goes where. There's find and zdb commands floating about to help create histograms of file sizes and determine the optimal recordsize for a given dataset.

Now, if I understand things right zfs uses variable recordsizes by default, up to the recordsize defined for the pool or dataset. So, say thats 128K, a file gets chopped up in 128K pieces and the final piece gets a smaller recordsize to accomodate for the leftover piece. I hope I got that right or I've been lying to a lot of people haha!

Assuming I got that right, my question then is will those variable records (when <= small block size) go to the special vdev as well?

The reason I ask is that I am duplicating (zfs send/recv) a dataset with 1M recordsizes (while preserving recordsize), and my special vdev with small block size set to 128K is filling up. This dataset contains almost exclusively 2 filesizes:

1k   | 12
8k   | 9826
64k  | 2
8M   | 1
64M  | 2
128M | 10
256M | 9810

These are backups, each data file apparently has an 8k sidecar file.

So, well I'm not sure what to make of this so I figured I'd ask and see if someone can help me understand?

Thanks!

[amotin]

special_small_blocks property is applied to a block size after compression. So it may be a "variable record size" for a small file, or just a well compressed large block of a large file. Anything with on-disk size below the specified threshold will be written to special vdev.

[Hr46ph]

Still something is unclear. I think I understand what you're saying yet when the above zfs sync of dataset with 1M recordsize ended up largely on the special vdev, I started over after setting special_small_blocks=64k. Allocation on the special vdev went down to below 100G and after syncing a ~500GB dataset with 64k recordsize (rsync backups of linux hosts) allocation on the special vdev rose with only ~50G.

Can I conclude then that recordsize has got nothing to do with how small_blocks are determined?

From your explanation I gathered a 1M record could end up < 128KB and thus be written to the special vdev. How come then that a 500G 64K recordsize only ~10% ends up on the special vdev (and some of that is metadata)?

[Hr46ph]

I suspect I found the culprit ... can someone please confirm if this is correct?

I had all my recordsizes at default 128K. I checked block sizes with zdb -Lbbbs and a find-command. Decided to set some to 1M, some to 64K and leave most to 128K.

Whatever was written with 128K will remain 128K after a change to 1M. So the question here is, when using zfs-send will it use the actual recordsize or the changed 1M?

Its the only thing I can think of right now.

Using sanoid with --preserve-recordsize might also be unpredictive, I don't think its a zfs-send option.

[rincebrain]

zfs send will almost never change record sizes.

If you don't use -L, records above 128k will get chopped into 128k slices.

[Hr46ph]

Thanks. Its getting more complicated now.

So I can't use send/recv to make changes to record size? If it was 128k and changed to 1M, the received dataset might be set to 1M but all the records that were 128k will still be 128k?

To change the record size of a dataset (and force small blocks to special vdev) I am forced to use something like rsync? I cannot use zfs send/recv? Basically I will loose all my snapshots?

[rincebrain]

Correct.

[Hr46ph]

Thank you!

I guess I need to turn down my small_block_size 😁 .

[rincebrain]

Also you're wrong, all records are logically the same size after you exceed one record in a file; they might physically end up smaller if you have compression enabled, but logically it's still 128k or 1M or whatever.

[IvanVolosyuk]

Just to clarify a bit. Record size went you created a file doesn't change when you change the dataset recordsize property. It is used to cut the file logically to recordsize chunks. Each chunk (record) is compressed and stored on disk as a block (as a sequence of (1<<ashift)-sized ZFS-sectors). ZFS send doesn't change the recordsize of the file (or can break large records into 128k chunks without -L option). The compression of the records can change though as well as raidz/draid and ashift overhead.

[rincebrain]

That's also inaccurate.

recordsize is an upper bound per file. Once a file takes 2 or more records, the recordsize becomes fixed for the lifetime of the file, but otherwise, it will keep scaling the size up every time you would exceed the current record size until it would exceed the current recordsize property to doso. then it makes another record.