How to deal with NVMe SSDs reporting huge block size?

[zrav]

We recently started using Micron 7450 NVMe SSDs but are now facing the issue that they report a huge native block size of 256k and ZFS complains about the mismatch. Obviously I can't and wouldn't want to use ashift=18.

root@server:~# zpool status
  pool: dozer
 state: ONLINE
status: One or more devices are configured to use a non-native block size.
        Expect reduced performance.
action: Replace affected devices with devices that support the
        configured block size, or migrate data to a properly configured
        pool.
config:

        NAME        STATE     READ WRITE CKSUM
        dozer       ONLINE       0     0     0
          nvme0n1   ONLINE       0     0     0  block size: 4096B configured, 262144B native
		  
root@server:~# nvme list
Node                  SN                   Model                                    Namespace Usage                      Format           FW Rev
--------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1          22263AAAAAAA         Micron_7450_MTFDKBA960TFR                1           0.00   B / 950.19  GB    512   B +  0 B   E2MU110

root@server:~# lsblk -t
NAME          ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED       RQ-SIZE  RA WSAME
nvme0n1               0 262144 262144  262144     512    0 none           1023 512    0B

Is there a workaround? As we use the pool status to monitor our ZFS pools, we'd need to silence this particular warning to have a clean output.

[amotin]

You should be able to silence it by setting ZPOOL_STATUS_NON_NATIVE_ASHIFT_IGNORE environment variable. I think ZFS should probably stop complaining about ashifts it can not use (above ASHIFT_MAX). I hope space allocation or write aggregation one day make use of the large values, but it is not a way for ashift to go.

[amotin]

We do not know FTL organization of this SSD, so we can only hope it makes sense to align to this value. Same time SSD has no idea what file system using it can or can't do, so it can only report whatever optimization is sees possible and hope it will be useful.

[slavanap]

Just a comment here. It may have sense to amend that list https://github.com/openzfs/zfs/blob/master/cmd/zpool/os/linux/zpool_vdev_os.c#L98

[amotin]

I think most of devices in that list do not report their physical sector size while actually have one. Overriding size explicitly reported by the device (and supposedly present) just because we can not use it at this point must have very good reason. I would not like to create a precedent.

[richardm1]

Would it be useful to benchmark such drives at various ashifts in an effort to tease out the underlying FTL organization? Perhaps by lowering ashift one increment at a time until performance falls off a cliff?

[amotin]

It may be not as obvious as for 512e HDDs, where any misaligned request is times slower. It may affect some metadata storage efficiency or something else noticeable only under significant stress, overloading device's internal RAM. Though I am not a firmware engineer, may only guess.