special_small_blocks expected behaviour?

[natankeddem]

I setup a brand new pool with 3x HDD in raidz and a mirror special device with 2x SSD. I set a recordsize of 4M and a special_small_blocks value of 2M. I start filling the pool which is being utilized by a Proxmox Backup Server to store data.

I expected for the zdb histogram 2M asize cumulative value to equal the data stored on my special device plus metadata. It appears like in fact the 1M value is being stored, does this look correct? I saw in earlier versions there might have been a < vs <= change but that was a while back and I am using a pretty recent version of zfs I believe. This might just be a misunderstanding on my part but I thought the histogram would show values of blocks and those blocks should be the same as what I would select for values special_small_blocks for items which are equal to and less than in size for storage to the special device.
zdb histogram

     1M:   190G
     2M:   397G

zfs special device stored data

  mirror-1   191G 

root@computer:~# zfs --version
zfs-2.1.12-pve1
zfs-kmod-2.1.12-pve1

root@computer:~# zpool history tank
History for 'tank':
2023-09-09.14:17:05 zpool create -o ashift=12 tank raidz /dev/sdf /dev/sdg /dev/sdh
2023-09-09.14:17:17 zpool add tank -o ashift=12 special mirror /dev/sdc /dev/sdd -f
2023-09-09.14:17:36 zfs set recordsize=4M tank
2023-09-09.14:17:45 zfs set special_small_blocks=2M tank
2023-09-09.14:17:56 zfs set mountpoint=/mnt/datastore/tank tank

root@computer:~# zpool iostat -v tank
              capacity     operations     bandwidth 
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tank        2.48T  37.2T     77    407   349K   210M
  raidz1-0  2.29T  35.9T      0    103  24.0K   178M
    sdf         -      -      0     34  6.86K  59.5M
    sdg         -      -      0     34  8.00K  59.5M
    sdh         -      -      0     34  9.10K  59.5M
special         -      -      -      -      -      -
  mirror-1   191G  1.27T     77    303   325K  31.8M
    sdc         -      -     38    152   163K  15.9M
    sdd         -      -     38    151   162K  15.9M
----------  -----  -----  -----  -----  -----  -----

root@computer:~# zdb -Lbbb tank

Traversing all blocks ...

2.38T completed (106294MB/s) estimated time remaining: 0hr 00min 00sec        
        bp count:                922846
        ganged count:                 0
        bp logical:       1910654520832      avg: 2070393
        bp physical:      1883945486848      avg: 2041451     compression:   1.01
        bp allocated:     2726828064768      avg: 2954802     compression:   0.70
        bp deduped:                   0    ref>1:      0   deduplication:   1.00
        Normal class:     2508660408320     used:  5.98%
        Special class      204123787264     used: 12.78%
        Embedded log class              0     used:  0.00%

        additional, non-pointer bps of type 0:        161
         number of (compressed) bytes:  number of bps
                         27:    113 ****************************************
                         28:     30 ***********
                         29:      0 
                         30:      0 
                         31:      0 
                         32:      1 *
                         33:      0 
                         34:      0 
                         35:      0 
                         36:      0 
                         37:      0 
                         38:      0 
                         39:      0 
                         40:      4 **
                         41:      0 
                         42:      0 
                         43:      0 
                         44:      0 
                         45:      1 *
                         46:      0 
                         47:      0 
                         48:      0 
                         49:      0 
                         50:      2 *
                         51:      0 
                         52:      0 
                         53:      0 
                         54:      0 
                         55:      0 
                         56:      0 
                         57:      0 
                         58:      0 
                         59:      0 
                         60:      0 
                         61:      0 
                         62:      0 
                         63:      0 
                         64:      0 
                         65:      1 *
                         66:      0 
                         67:      3 **
                         68:      0 
                         69:      0 
                         70:      0 
                         71:      0 
                         72:      0 
                         73:      0 
                         74:      0 
                         75:      1 *
                         76:      0 
                         77:      1 *
                         78:      0 
                         79:      0 
                         80:      0 
                         81:      0 
                         82:      0 
                         83:      0 
                         84:      0 
                         85:      1 *
                         86:      0 
                         87:      0 
                         88:      0 
                         89:      0 
                         90:      0 
                         91:      0 
                         92:      0 
                         93:      0 
                         94:      0 
                         95:      0 
                         96:      0 
                         97:      1 *
                         98:      0 
                         99:      0 
                        100:      1 *
                        101:      0 
                        102:      0 
                        103:      0 
                        104:      0 
                        105:      0 
                        106:      0 
                        107:      0 
                        108:      0 
                        109:      1 *
        Dittoed blocks on same vdev: 342949

Blocks  LSIZE   PSIZE   ASIZE     avg    comp   %Total  Type
     -      -       -       -       -       -        -  unallocated
     2    32K      8K     48K     24K    4.00     0.00  object directory
     1   128K      4K     12K     12K   32.00     0.00      L1 object array
     7  3.50K   3.50K     84K     12K    1.00     0.00      L0 object array
     8   132K   7.50K     96K     12K   17.53     0.00  object array
     1    16K      4K     24K     24K    4.00     0.00  packed nvlist
     -      -       -       -       -       -        -  packed nvlist size
     -      -       -       -       -       -        -  bpobj
     -      -       -       -       -       -        -  bpobj header
     -      -       -       -       -       -        -  SPA space map header
    39   624K    156K    468K     12K    4.00     0.00      L1 SPA space map
   513  64.1M   8.54M   25.6M   51.1K    7.51     0.00      L0 SPA space map
   552  64.7M   8.69M   26.1M   48.4K    7.45     0.00  SPA space map
     1    36K     36K     56K     56K    1.00     0.00  ZIL intent log
     1   128K      4K      8K      8K   32.00     0.00      L5 DMU dnode
     1   128K      4K      8K      8K   32.00     0.00      L4 DMU dnode
     1   128K      4K      8K      8K   32.00     0.00      L3 DMU dnode
     1   128K      4K      8K      8K   32.00     0.00      L2 DMU dnode
    16     2M    736K   1.45M   92.8K    2.78     0.00      L1 DMU dnode
 14.8K   237M   59.3M    119M   8.05K    4.00     0.00      L0 DMU dnode
 14.8K   240M   60.1M    121M   8.14K    3.99     0.00  DMU dnode
     2     8K      8K     20K     10K    1.00     0.00  DMU objset
     -      -       -       -       -       -        -  DSL directory
     -      -       -       -       -       -        -  DSL directory child map
     -      -       -       -       -       -        -  DSL dataset snap map
     5  33.5K      8K     24K   4.80K    4.19     0.00  DSL props
     -      -       -       -       -       -        -  DSL dataset
     -      -       -       -       -       -        -  ZFS znode
     -      -       -       -       -       -        -  ZFS V0 ACL
  127K  15.9G    509M   1019M      8K   32.00     0.04      L1 ZFS plain file
  566K  1.71T   1.71T   2.48T   4.48M    1.00    99.89      L0 ZFS plain file
  693K  1.73T   1.71T   2.48T   3.66M    1.01    99.93  ZFS plain file
 63.9K  7.98G    256M    511M      8K   32.00     0.02      L1 ZFS directory
  128K  2.00G    529M   1.03G   8.23K    3.88     0.04      L0 ZFS directory
  192K  9.99G    784M   1.53G   8.16K   13.04     0.06  ZFS directory
     1    512     512     16K     16K    1.00     0.00  ZFS master node
     -      -       -       -       -       -        -  ZFS delete queue
     -      -       -       -       -       -        -  zvol object
     -      -       -       -       -       -        -  zvol prop
     -      -       -       -       -       -        -  other uint8[]
     -      -       -       -       -       -        -  other uint64[]
     -      -       -       -       -       -        -  other ZAP
     -      -       -       -       -       -        -  persistent error log
     1   128K      4K     12K     12K   32.00     0.00  SPA history
     -      -       -       -       -       -        -  SPA history offsets
     -      -       -       -       -       -        -  Pool properties
     -      -       -       -       -       -        -  DSL permissions
     -      -       -       -       -       -        -  ZFS ACL
     -      -       -       -       -       -        -  ZFS SYSACL
     -      -       -       -       -       -        -  FUID table
     -      -       -       -       -       -        -  FUID table size
     -      -       -       -       -       -        -  DSL dataset next clones
     -      -       -       -       -       -        -  scan work queue
     -      -       -       -       -       -        -  ZFS user/group/project used
     -      -       -       -       -       -        -  ZFS user/group/project quota
     -      -       -       -       -       -        -  snapshot refcount tags
     -      -       -       -       -       -        -  DDT ZAP algorithm
     -      -       -       -       -       -        -  DDT statistics
     -      -       -       -       -       -        -  System attributes
     -      -       -       -       -       -        -  SA master node
     1  1.50K   1.50K     16K     16K    1.00     0.00  SA attr registration
     2    32K      8K     16K      8K    4.00     0.00  SA attr layouts
     -      -       -       -       -       -        -  scan translations
     -      -       -       -       -       -        -  deduplicated block
     -      -       -       -       -       -        -  DSL deadlist map
     -      -       -       -       -       -        -  DSL deadlist map hdr
     -      -       -       -       -       -        -  DSL dir clones
     -      -       -       -       -       -        -  bpobj subobj
     -      -       -       -       -       -        -  deferred free
     -      -       -       -       -       -        -  dedup ditto
    18   334K     38K    216K     12K    8.78     0.00  other
     1   128K      4K      8K      8K   32.00     0.00      L5 Total
     1   128K      4K      8K      8K   32.00     0.00      L4 Total
     1   128K      4K      8K      8K   32.00     0.00      L3 Total
     1   128K      4K      8K      8K   32.00     0.00      L2 Total
  191K  23.9G    766M   1.50G   8.01K   31.97     0.06      L1 Total
  710K  1.71T   1.71T   2.48T   3.58M    1.00    99.94      L0 Total
  901K  1.74T   1.71T   2.48T   2.82M    1.01   100.00  Total

Block Size Histogram

  block   psize                lsize                asize
   size   Count   Size   Cum.  Count   Size   Cum.  Count   Size   Cum.
    512:    151  75.5K  75.5K    151  75.5K  75.5K      0      0      0
     1K:     94   106K   182K     94   106K   182K      0      0      0
     2K:     37  94.5K   276K     37  94.5K   276K      0      0      0
     4K:   331K  1.29G  1.29G     56   272K   548K    262  1.02M  1.02M
     8K:  4.02K  32.3M  1.32G     60   632K  1.15M   331K  2.59G  2.59G
    16K:    119  2.76M  1.33G   143K  2.24G  2.24G  4.05K  65.9M  2.65G
    32K:    479  22.7M  1.35G    452  21.4M  2.26G    423  19.1M  2.67G
    64K:    703  56.3M  1.40G    634  50.3M  2.31G    717  56.2M  2.73G
   128K:  3.57K   561M  1.95G   195K  24.5G  26.8G  3.59K   568M  3.28G
   256K:  1.67K   632M  2.57G  1.67K   632M  27.4G  1.72K   644M  3.91G
   512K:  3.64K  2.85G  5.42G  3.64K  2.85G  30.3G  3.53K  2.74G  6.65G
     1M:   125K   184G   189G   125K   184G   214G   124K   183G   190G
     2M:   142K   411G   600G   142K   411G   625G  59.7K   207G   397G
     4M:   289K  1.13T  1.71T   289K  1.13T  1.74T   372K  2.09T  2.48T
     8M:      0      0  1.71T      0      0  1.74T      0      0  2.48T
    16M:      0      0  1.71T      0      0  1.74T      0      0  2.48T

[kftsehk]

it depends on the ZFS "flavour" you used, I does see

#15388

merged before covid, and it says, if you type

zfs set 2>&1 | grep special

... zero or 512 to max_recordsize, power of 2

but on e.g. archlinux zfs-dkms, I still see this

zfs --version
zfs-2.1.13-1
zfs-kmod-2.1.13-1

zfs set 2>&1 | grep special
        special_small_blocks YES      YES   zero or 512 to 1M, power of 2

There seem to be a divergence between my package manager, aur/zfs-dkms 2.1.13-1 (+156 2.50) (Installed) vs what seem should be released 2 years ago.

I wouldn't worry of 1M in your case, as you have shown even with 1M record size

• planned capacity is 40TB raidz:1.25TB special, around 20:1
• on utility side: 2.3TB on raidz:190GB on special, around ~ 20: 2

already using twice planned capacity on special vdev than planned, if your data isn't skewed in some particular way now compared to future workloads, u run out of special device space when the pool is half filled (i.e. if you have 23TB data on raidz, then the ratio says you end up 1.9TB on special, which you don't have) and block spill back into raidz anyway even with 1M record size

[natankeddem]

Thank you for the insight, I appreciate it.