Intel QAT compression fails with ZFS 2.0.0

[fstrati70]

System information:

Distribution Centos 8.2
Stock kernel:

[root@dellqat zfs_latest]# uname -a
Linux dellqat 4.18.0-193.19.1.el8_2.x86_64 #1 SMP Mon Sep 14 14:37:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@dellqat zfs_latest]# modinfo zfs | grep -iw version
version:        2.0.0-1
[root@dellqat zfs_latest]# modinfo spl | grep -iw version
version:        2.0.0-1

ZFS 2.0.0 fails to use Intel QAT compression.
QAT and ZFS compiled from sources:

QAT:

qat1.7.l.4.11.0-00001 (latest from intel)

./configure --enable-icp-trace --enable-icp-debug --enable-icp-log-syslog --enable-kapi
make
make install

ZFS:

[root@dellqat zfs_latest]# git status
On branch zfs-2.0-release
Your branch is up to date with 'origin/zfs-2.0-release'.

nothing to commit, working tree clean

export ICP_ROOT=/opt/A3C/qat1.7.l.4.11.0-00001
./configure --with-qat=/opt/A3C/qat1.7.l.4.11.0-00001
make
make install
ldconfig

(tried also various reboots...)

[root@dellqat ~]# lsmod | grep qat
qat_api               634880  2 zfs
qat_dh895xcc           20480  0
intel_qat             249856  3 qat_api,usdm_drv,qat_dh895xcc
uio                    20480  1 intel_qat
[root@dellqat ~]# lsmod | grep zfs
zfs                  4481024  1
zunicode              335872  1 zfs
zzstd                 507904  1 zfs
qat_api               634880  2 zfs
zlua                  176128  1 zfs
zcommon                94208  1 zfs
znvpair                90112  2 zfs,zcommon
zavl                   16384  1 zfs
icp                   323584  1 zfs
spl                   110592  6 zfs,icp,zzstd,znvpair,zcommon,zavl

zpool create -f -m /mnt/test test /dev/sdb /dev/sdc /dev/sdd

[root@dellqat ~]# zfs set compression=gzip test
[root@dellqat ~]# zfs get all test
NAME  PROPERTY              VALUE                  SOURCE
test  type                  filesystem             -
test  creation              mar dic 15 10:58 2020  -
test  used                  333K                   -
test  available             2.63T                  -
test  referenced            24K                    -
test  compressratio         1.00x                  -
test  mounted               yes                    -
test  quota                 none                   default
test  reservation           none                   default
test  recordsize            128K                   default
test  mountpoint            /mnt/test              local
test  sharenfs              off                    default
test  checksum              on                     default
test  compression           gzip                   local
test  atime                 on                     default
test  devices               on                     default
test  exec                  on                     default
test  setuid                on                     default
test  readonly              off                    default
test  zoned                 off                    default
test  snapdir               hidden                 default
test  aclmode               discard                default
test  aclinherit            restricted             default
test  createtxg             1                      -
test  canmount              on                     default
test  xattr                 on                     default
test  copies                1                      default
test  version               5                      -
test  utf8only              off                    -
test  normalization         none                   -
test  casesensitivity       sensitive              -
test  vscan                 off                    default
test  nbmand                off                    default
test  sharesmb              off                    default
test  refquota              none                   default
test  refreservation        none                   default
test  guid                  3091558646933016362    -
test  primarycache          all                    default
test  secondarycache        all                    default
test  usedbysnapshots       0B                     -
test  usedbydataset         24K                    -
test  usedbychildren        309K                   -
test  usedbyrefreservation  0B                     -
test  logbias               latency                default
test  objsetid              54                     -
test  dedup                 off                    default
test  mlslabel              none                   default
test  sync                  standard               default
test  dnodesize             legacy                 default
test  refcompressratio      1.00x                  -
test  written               24K                    -
test  logicalused           115K                   -
test  logicalreferenced     12K                    -
test  volmode               default                default
test  filesystem_limit      none                   default
test  snapshot_limit        none                   default
test  filesystem_count      none                   default
test  snapshot_count        none                   default
test  snapdev               hidden                 default
test  acltype               off                    default
test  context               none                   default
test  fscontext             none                   default
test  defcontext            none                   default
test  rootcontext           none                   default
test  relatime              off                    default
test  redundant_metadata    all                    default
test  overlay               on                     default
test  encryption            off                    default
test  keylocation           none                   default
test  keyformat             none                   default
test  pbkdf2iters           0                      default
test  special_small_blocks  0                      default

[root@dellqat /]# cat /proc/spl/kstat/zfs/qat 
20 1 0x01 17 4624 14088835193 61490940492455
name                            type data
comp_requests                   4    0
comp_total_in_bytes             4    0
comp_total_out_bytes            4    0
decomp_requests                 4    0
decomp_total_in_bytes           4    0
decomp_total_out_bytes          4    0
dc_fails                        4    0
encrypt_requests                4    0
encrypt_total_in_bytes          4    0
encrypt_total_out_bytes         4    0
decrypt_requests                4    0
decrypt_total_in_bytes          4    0
decrypt_total_out_bytes         4    0
crypt_fails                     4    0
cksum_requests                  4    0
cksum_total_in_bytes            4    0
cksum_fails                     4    0

Tested with various methods: dd, standard cp & mv, iozone

Please note the strange:
test compressratio 1.00x -

Seems compression is not there at all.

[fstrati70]

Also
[root@dellqat zfs_latest]# adf_ctl status
Checking status of all devices.
There is 1 QAT acceleration device(s) in the system:
qat_dev0 - type: dh895xcc, inst_id: 0, node_id: 1, bsf: 0000:85:00.0, #accel: 6 #engines: 12 state: up

[fstrati70]

It is the module loading order,
after reboot, issuing

modprobe -r zfs;
modprobe zfs;

solves the problem, in the sense that the qat module is used correctly in compression.

Hence, the question arise about the module loading ordering, because zfs seems to load before qat
and qat is initialised in userspace by the qat_service.

Do you have any suggestion on how to proceed?
Maybe with a custom initramfs?
Does zfs modify the initramfs?

[behlendorf]

You should be able to use the modprobe.d softdep option to create a soft dependency on the qat module to request it be loaded before the zfs module. Like so:

cat /etc/modprobe.d/zfs.conf
softdep zfs pre: qat

https://man7.org/linux/man-pages/man5/modprobe.d.5.html

[fstrati70]

Thanks for the answer, but qat needs an userspace program "adf_ctl" in order to start its engines.
Hence it is not only a question of module ordering, adf_ctl must be run after loading the qat modules
and prior to load zfs. That is why I was looking into a custom initramfs.
It is kinda strange though... not clear about the role of the adf_ctl program...

[behlendorf]

I see. You might be able to additionally leverage the modprobe.d install option to run "adf_ctl" at the right time.

       install modulename command...
           This command instructs modprobe to run your command instead of
           inserting the module in the kernel as normal. The command can be
           any shell command: this allows you to do any kind of complex
           processing you might wish.
           ...

[fstrati70]

I sort of got it working with a custom initramfs with dracut...

You must exclude the dracut modules for zfs:
02zfsexpandknowledge and 90zfs.

You then add a custom module for qat:

[root@dellqat ~]# cd /usr/lib/dracut/modules.d/
[root@dellqat modules.d]# mkdir 89qat

inside 89qat you put two files:

[root@dellqat modules.d]# cd 89qat/
[root@dellqat 89qat]# cat module-setup.sh 
#!/usr/bin/env bash

check() {
	return 0
}

depends() {
	return 0
}

installkernel() {
	instmods qat_dh895xcc
	instmods qat_api
	instmods usdm_drv
	instmods intel_qat
	instmods uio
}

install() {
	inst_hook pre-mount 10 "${moddir}/qat_start.sh"
	inst /etc/dh895xcc_dev0.conf /etc/dh895xcc_dev0.conf
	inst /usr/local/bin/adf_ctl /sbin/adf_ctl
	inst /usr/bin/sleep /sbin/sleep
}

and

[root@dellqat 89qat]# cat qat_start.sh
#!/bin/sh

modprobe -r qat_api;
/sbin/sleep 1;
modprobe -r qat_dh895xcc;
/sbin/sleep 1;
modprobe qat_dh895xcc;
/sbin/sleep 1;
modprobe qat_api;
/sbin/sleep 1;
/sbin/adf_ctl restart 1>/dev/null 2>&1;
/sbin/sleep 1;

both files shall be chmod 755.

you then rebuild the image with dracut:
dracut foobar.img $(uname -r) --force -v

which create the "foobar.img" initramfs image for the current kernel.

I don't know why but the modules don't load in the correct order
if you keep also the zfs-dracut modules: 02zfsexpandknowledge and 90zfs.

Maybe someone with more experience with dracut may help?

[fstrati70]

@behlendorf
If I understand well modprobe.d conf files will be processed at boot time after initial setup from initramfs.
That is why I was looking for a custom initramfs with dracut: to be able to integrate with zfs-dracut
so as to have also root partitions with zfs and qat

[fstrati70]

Also, it is unclear when the zfs module will be loaded at boot time, from dmesg I see it after initramfs, but have you any pointer on the way it is configured to load zfs module at boot time ? For the Centos 8 of course. It seems it depends on the distribution the way the module loading process happens

[behlendorf]

Normally the way the module loading happens is through the udev rule 90-zfs.rules. If a block device with a ZFS label is detected the modules will be automatically loaded. The modules should also be automatically loaded if a process tries to access the /dev/zfs device, a6cc975.

I'd like to be more help, but I'm not that familiar with the specifics of dracut.

[bghira]

https://www.computerhope.com/unix/depmod.htm

depmod creates a list of module dependencies by reading each module under /lib/modules/kernel-release and determining what symbols it exports and what symbols it needs. By default, this list is written to modules.dep, and a binary hashed version named modules.dep.bin, in the same directory. If file names are given on the command line, only those modules are examined (which is rarely useful unless all modules are listed).

depmod also creates a list of symbols provided by modules in the file named modules.symbols and its binary hashed version, modules.symbols.bin.

Finally, depmod outputs a file named modules.devname if modules supply special device names (devname) that should be populated in /dev on boot (by a utility such as udev).

so it sounds like depmod isn't finding some dependencies because some symbols aren't matching up?

[sempervictus]

@behlendorf - i just ran into this, and my ZFS is built-in... is there any way to trigger use of QAT at runtime (safely)?

[sempervictus]

Trying to address this by building in the specific QAT module for the HW in that host, but no guarantees of load-order here so not sure how this will go.
Is there anything in dmesg i can look for to signify that compression & crypto are being handled by qat?

[allegro0132]

I think we could modify the udev rule 90-zfs.rules for OpenZFS build with QAT support, loading qat_api and spl before ZFS label is detected.

sed -i 's|/sbin/modprobe zfs|/sbin/modprobe qat_api\&\&/sbin/modprobe spl\&\&/sbin/modprobe zfs|g' /usr/lib/udev/rules.d/90-zfs.rules

I tested it in zfs-2.2.3 and QAT works smoothly during linux booting.