Buffer I/O error on dev zd0, logical block ..., async page read

[knyar]

I have been running OpenZFS on a physical machine with Debian 11 for slightly more than a year without any issues. Recently I started a qemu virtual machine via libvirt, and decided to use a couple of zvol volumes for this VMs block devices.

As soon as I started using the zvol, I noticed the following messages reported by the kernel:

Buffer I/O error on dev zd0, logical block 42025802, async page read

These appear regularly in short bursts, supposedly when the VM is trying to use respective blocks. When this happens, I am seeing CKSUM errors incremented in zpool status.

  pool: serverh
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: resilvered 0B in 02:02:41 with 0 errors on Sat Dec 17 17:29:15 2022
config:

        NAME                                      STATE     READ WRITE CKSUM
        serverh                                   ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            3c8b2333-7b7a-46f8-b515-d9163679a83b  ONLINE       0     0     4
            973f071f-ecf4-47da-a2c7-243d918cbbc7  ONLINE       0     0     4
        special
          mirror-3                                ONLINE       0     0     0
            84ea3d26-04                           ONLINE       0     0     0
            61b88409-04                           ONLINE       0     0     0
        logs
          mirror-2                                ONLINE       0     0     0
            84ea3d26-01                           ONLINE       0     0     0
            61b88409-01                           ONLINE       0     0     0
        cache
          84ea3d26-03                             ONLINE       0     0     0
          61b88409-03                             ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        serverh/virt-windowsvm1:<0x1>

The pool that this volume is part of is built on a mirror composed of two hard drive partitions, and when these buffer I/O errors happen, I am not seeing any I/O errors from the underlying device reported by the kernel, and SMART counters for both hard drives show no errors.

I tried setting sync=always for these zvols, but the errors still kept appearing.

I tried upgrading ZFS and the kernel to the latest versions available in Debian backports repository, but I keep seeing these errors even on the newer versions.

Today I decided to stop using zvols and tried to migrate data from these volumes to qemu image files, but qemu-img convert process failed because of an I/O error.

More details about my environment (zpool history, SMART output, list of blocks this is happening for) are in this log.

At this point this seems like a bug in OpenZFS because I am not really seeing any I/O errors lower in the stack, but I am creating this as a Q&A discussion (rather than a Github issue) just in case I am missing something obvious. Any thoughts?

[rincebrain]

What does zpool events -v have logged when these checksum errors show up?

Are you running zpool clear at any point, because it should definitely be showing those bubbling up past the individual devices.

If you migrate one of the zvols to not have encryption enabled, does the problem still keep happening? (To be clear, it'd still be a bug if that were true, just asking to narrow down what's wrong.)

It might be interesting to see whether 2.1.7 resolves this for you, though I can't think of any fixes that would be germane.

I should remark that 2.1.6 doesn't claim to work on 6.0, so it's not impossible that could be a problem for you, though in practice I don't see any explicit 6.0 support fixes in 2.1.7 other than listing it as supported...

[knyar]

Taking a closer look at zfs events I noticed ereport.fs.zfs.checksum that appear alongside ereport.fs.zfs.data errors I shared above: https://knyar.net/paste/89ff43ef

[knyar]

Right now my zfs events contains checksum errors for 3 blocks, and they share the same characteristics:

• Both devices return the same actual checksum
• Checksums differ in the second to last byte
• Most of them are not single-bit flips.

The machine runs non-ECC RAM, so I suppose these could be hardware-related issues even if they are not single-bit flips? It is a bit odd that I have not seen checksum errors on regular ZFS volumes (or btrfs) on this machine for years, and these appeared shortly after I created a first zvol, but I guess it could be a coincidence.

[rincebrain]

It's also very strange because even if that's the case, it should be bubbling those error counters up to higher levels if they're causing user-visible errors, unless I really misunderstand how this is supposed to work.

To be clear, my suggestion to try 2.1.7 was not because I thought 2.1.6 specifically was broken, but just to be sure no fixes that are already in a release ameliorated the problem, as that would make it rather frustrating to root cause only to find we already did that and fixed it...

[knyar]

I've upgraded to 2.1.7, and still see these checksum errors.

        NAME                                      STATE     READ WRITE CKSUM
        serverh                                   ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            3c8b2333-7b7a-46f8-b515-d9163679a83b  ONLINE       0     0   306
            973f071f-ecf4-47da-a2c7-243d918cbbc7  ONLINE       0     0   306
        special
          mirror-3                                ONLINE       0     0     0
            84ea3d26-04                           ONLINE       0     0     0
            61b88409-04                           ONLINE       0     0     0
        logs
          mirror-2                                ONLINE       0     0     0
            84ea3d26-01                           ONLINE       0     0     0
            61b88409-01                           ONLINE       0     0     0
        cache
          84ea3d26-03                             ONLINE       0     0     0
          61b88409-03                             ONLINE       0     0     0

Event log since reboot: https://knyar.net/paste/cb9682ea

[knyar]

Rebooting the machine with the memtest parameter set, I confirmed that the machine had memory issues that were likely causing this.

Dec 19 15:21:00 server kernel: [   21.770978] Bad RAM detected. Use memtest86+ to perform a thorough test
...
Dec 19 15:21:00 server kernel: [   21.771025]   0000000000000000 bad mem addr 0x00000005ff987ff8 - 0x00000005ff988000 reserved