zpool create using file as device

[0xzfs]

Hello devs,

I was trying to make zfs filesystem in a file and I came up with something weird regarding allocated space on a zpool device.
Basically, 10-20% of the space is missing (depending on the underlying file size), when creating zpool in a file (compared to HDD or SDD), already at the zpool level.

I'm attaching terminal logs, to better understand the situation. For comparison, I used ext4 in parallel.

Thanks in advance for clues and help, much appreciated!

**

a@work:/tmp/test$ sudo mkdir /mnt/ext4
a@work:/tmp/test$ sudo mkdir /mnt/zfs

** allocating 5G space
a@work:/tmp$ cd test/
a@work:/tmp/test$ fallocate -l 5G ext4.img
a@work:/tmp/test$ fallocate -l 5G zfs.img

@work:/tmp/test$ ls -sh
total 10G
5,0G ext4.img  5,0G zfs.img

** creating ext4 fs
a@work:/tmp/test$ mkfs.ext4 ext4.img 
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done                            
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: 0914814f-d775-4a67-80a9-9146f5b93bb9
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

** setup loopback
a@work:/tmp/test$ sudo losetup -f --show ext4.img 
/dev/loop28

** mounting image
a@work:/tmp/test$ sudo mount /dev/loop28 /mnt/ext4
a@work:/tmp/test$ df -h | grep ext4
/dev/loop28                  4,9G   20M  4,6G   1% /mnt/ext4

** OKAY, missing 400M because of default 5% reserved space

** quick fix
a@work:/tmp/test$ sudo tune2fs -m 0 /dev/loop28
tune2fs 1.45.5 (07-Jan-2020)
Setting reserved blocks percentage to 0% (0 blocks)

a@work:/tmp/test$ df -h | grep ext4
/dev/loop28                  4,9G   20M  4,9G   1% /mnt/ext4

** OKAY, excellent!

** now lets do the same, this time with ZFS

a@work:/tmp/test$ sudo zpool create testpool /tmp/test/zfs.img 
a@work:/tmp/test$ df -h | grep test
testpool                     4,4G  128K  4,4G   1% /testpool
a@work:/tmp/test$ zpool show -all

a@work:/tmp/test$ sudo zpool list
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
testpool  4,50G   116K  4,50G        -         -     0%     0%  1.00x    ONLINE  -

a@work:/tmp/test$ sudo zfs list
NAME       USED  AVAIL     REFER  MOUNTPOINT
testpool  91,5K  4,36G       24K  /testpool

** missing 600M+ (12% of 5G) !! **
** What is happening here? :) where is 600M or how to reduce that??
** This can be fairly simply reproduced on any Ubuntu unity 2020 with latest zfs tools

a@work:/tmp/test$ zfs version
zfs-0.8.3-1ubuntu12.13
zfs-kmod-2.0.6-1ubuntu2

a@work:/tmp/test$ sudo zfs get all

NAME      PROPERTY              VALUE                  SOURCE
testpool  type                  filesystem             -
testpool  creation              pet mar  4  9:01 2022  -
testpool  used                  91,5K                  -
testpool  available             4,36G                  -
testpool  referenced            24K                    -
testpool  compressratio         1.00x                  -
testpool  mounted               yes                    -
testpool  quota                 none                   default
testpool  reservation           none                   default
testpool  recordsize            128K                   default
testpool  mountpoint            /testpool              default
testpool  sharenfs              off                    default
testpool  checksum              on                     default
testpool  compression           off                    default
testpool  atime                 on                     default
testpool  devices               on                     default
testpool  exec                  on                     default
testpool  setuid                on                     default
testpool  readonly              off                    default
testpool  zoned                 off                    default
testpool  snapdir               hidden                 default
testpool  aclinherit            restricted             default
testpool  createtxg             1                      -
testpool  canmount              on                     default
testpool  xattr                 on                     default
testpool  copies                1                      default
testpool  version               5                      -
testpool  utf8only              off                    -
testpool  normalization         none                   -
testpool  casesensitivity       sensitive              -
testpool  vscan                 off                    default
testpool  nbmand                off                    default
testpool  sharesmb              off                    default
testpool  refquota              none                   default
testpool  refreservation        none                   default
testpool  guid                  2063925752334347009    -
testpool  primarycache          all                    default
testpool  secondarycache        all                    default
testpool  usedbysnapshots       0B                     -
testpool  usedbydataset         24K                    -
testpool  usedbychildren        67,5K                  -
testpool  usedbyrefreservation  0B                     -
testpool  logbias               latency                default
testpool  objsetid              54                     -
testpool  dedup                 off                    default
testpool  mlslabel              none                   default
testpool  sync                  standard               default
testpool  dnodesize             legacy                 default
testpool  refcompressratio      1.00x                  -
testpool  written               24K                    -
testpool  logicalused           34,5K                  -
testpool  logicalreferenced     12K                    -
testpool  volmode               default                default
testpool  filesystem_limit      none                   default
testpool  snapshot_limit        none                   default
testpool  filesystem_count      none                   default
testpool  snapshot_count        none                   default
testpool  snapdev               hidden                 default
testpool  acltype               off                    default
testpool  context               none                   default
testpool  fscontext             none                   default
testpool  defcontext            none                   default
testpool  rootcontext           none                   default
testpool  relatime              off                    default
testpool  redundant_metadata    all                    default
testpool  overlay               off                    default
testpool  encryption            off                    default
testpool  keylocation           none                   default
testpool  keyformat             none                   default
testpool  pbkdf2iters           0                      default
testpool  special_small_blocks  0                      default

a@work:/tmp/test$ sudo zpool get all
NAME      PROPERTY                       VALUE                          SOURCE
testpool  size                           4,50G                          -
testpool  capacity                       0%                             -
testpool  altroot                        -                              default
testpool  health                         ONLINE                         -
testpool  guid                           4639028818594675926            -
testpool  version                        -                              default
testpool  bootfs                         -                              default
testpool  delegation                     on                             default
testpool  autoreplace                    off                            default
testpool  cachefile                      -                              default
testpool  failmode                       wait                           default
testpool  listsnapshots                  off                            default
testpool  autoexpand                     off                            default
testpool  dedupditto                     0                              default
testpool  dedupratio                     1.00x                          -
testpool  free                           4,50G                          -
testpool  allocated                      116K                           -
testpool  readonly                       off                            -
testpool  ashift                         0                              default
testpool  comment                        -                              default
testpool  expandsize                     -                              -
testpool  freeing                        0                              -
testpool  fragmentation                  0%                             -
testpool  leaked                         0                              -
testpool  multihost                      off                            default
testpool  checkpoint                     -                              -
testpool  load_guid                      16211372470545943033           -
testpool  autotrim                       off                            default
testpool  feature@async_destroy          enabled                        local
testpool  feature@empty_bpobj            enabled                        local
testpool  feature@lz4_compress           active                         local
testpool  feature@multi_vdev_crash_dump  enabled                        local
testpool  feature@spacemap_histogram     active                         local
testpool  feature@enabled_txg            active                         local
testpool  feature@hole_birth             active                         local
testpool  feature@extensible_dataset     active                         local
testpool  feature@embedded_data          active                         local
testpool  feature@bookmarks              enabled                        local
testpool  feature@filesystem_limits      enabled                        local
testpool  feature@large_blocks           enabled                        local
testpool  feature@large_dnode            enabled                        local
testpool  feature@sha512                 enabled                        local
testpool  feature@skein                  enabled                        local
testpool  feature@edonr                  enabled                        local
testpool  feature@userobj_accounting     active                         local
testpool  feature@encryption             enabled                        local
testpool  feature@project_quota          active                         local
testpool  feature@device_removal         enabled                        local
testpool  feature@obsolete_counts        enabled                        local
testpool  feature@zpool_checkpoint       enabled                        local
testpool  feature@spacemap_v2            active                         local
testpool  feature@allocation_classes     enabled                        local
testpool  feature@resilver_defer         enabled                        local
testpool  feature@bookmark_v2            enabled                        local

[rincebrain]

You're using a mix of ZFS versions for userland and kernel, and you should stop using Ubuntu's broken setup. That's not related to your question, it's just a bad idea.

That said, I think you're observing a result of how ZFS allocates metaslabs - looking at zdb -MM, it's using 512 MB metaslabs for a 5GB file vdev, which means that there's going to be an area of (512 MB - however much it's using for fixed usage at the front and back of the pool) that's not covered, I think.

You could play with the zfs_vdev_default_ms_shift and similar tunables to adjust how it bins up the space - for example, setting it from the default of 29 (512 MB) to 25 (32 MB) results in:

NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
spaceme  4.97G   129K  4.97G        -         -     0%     0%  1.00x    ONLINE  -

Maybe something someone might want to consider reviewing and revising if it's not already done better in git or 2.1.

[0xzfs]

Thanks for the tip, yeah, unity seems to have broken setup, fixed in impish. Fixed it.

That is a great idea and seems like a solution, but I'm not sure where to set that - under zfs.conf or echoing to /sys/ ..?

Thanks again!

[rincebrain]

It's a module parameter, so /sys/module/zfs/parameters/... or in a file in /etc/modprobe.d or equivalent or ...

e: also, the mixed versions thing is just a thing Ubuntu always does whenever they ship an HWE kernel - they include the newer ZFS modules with it, but they've refused to provide any packaging for the newer userland, so you can end up with wildly differing versions and strange bugs that nobody else sees, and that's before things like the recent...incident...with 21.10 shipping a broken ZFS on release.

[0xzfs]

I see, let me play with it a bit:

A)
echo 25 > /sys/module/zfs/parameters/zfs_vdev_default_ms_shift

B)
echo "options zfs zfs_vdev_default_ms_shift=25" >> /etc/modprobe.d/zfs.conf

I'll get back, thanks a lot for now!

[0xzfs]

That fixed it!! That's just great, thanks a lot, I was really desperate with it !! 👍

It's beyond my zfs knowledge, but "metaslab reserved" space issue does not occur on say 4G external USB stick.

I intended to use the zfs within a file pretty much for some specific archiving purpose to get advantage of encryption/compression/snapshot, performance is not that important and the way I understand the metaslabs, I should be fine with a smaller one. Tho it's weird that can only be set per-system and not per-pool.

offtopic:
Regarding ubuntu, that's quite bad :) back in the days I was on gentoo, once CFLAGS, USE flags are optimized for your CPU and needs, it will run 30-50% faster than ubuntu, consume 20-30% less ram. But on old CPU-s compilation time was nightmare, package that just won't compile due to some obscure problem that the only result on google was 1 chinese site, 99% in mandarin except the error message. I might switch to gentoo or debian again soon.