how to use checkpoint to copy disk image live?

[aerusso]

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	testing
Linux Kernel	5.9.15 (host) and 5.10.19 (guest)
Architecture	amd64
ZFS Version	0.8.6 (host) and 2.0.3 (guest)

Describe the problem you're observing

I'm trying to get a valid image of a live zpool. I expected that zpool checkpoint should do this for me, but I'm at least running into some issue (perhaps a PEBCAK).

The pool in question has a single-vdev (~1TB). I want to capture an image of it, and import the image in a VM to diagnose a presumably unrelated problem. (I could bring the system down to single-user mode, unmount the pool, make the image, and then bring the computer back up, but I wanted to try something more elegant. I'm also not sure if I'm just doing something dumb right now.)

Describe how to reproduce the problem

On the host

qemu-img --connect=/dev/nbd0 /someimage
zpool checkpoint poolname
cat </dev/poolvdev >/dev/nbd0
zpool checkpoint -d poolname
qemu-img -d /dev/nbd0

(/dev/poolvdev is the disk not the partition on the disk).

Then, add the associated qcow2 to the VM as /dev/vdb.

On the guest, get txgid from zdb -k -l /dev/vdb1:

------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'poolname'
    state: 0
    txg: 12420925
    pool_guid: 1049664879909746186
    errata: 0
    hostid: 729205553
    hostname: 'myredactedhostname'
    top_guid: 6333517102995003485
    guid: 6333517102995003485
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 6333517102995003485
        path: '/dev/disk/by-id/nvme-eui.REDACTED-part1'
        whole_disk: 0
        metaslab_array: 131
        metaslab_shift: 33
        ashift: 12
        asize: 897120731136
        is_log: 0
        DTL: 33404
        create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
    labels = 0 1 2 3 

Finally:

zpool import -T 12420925 --rewind-to-checkpoint -f poolname

I get

cannot import 'poolname': one or more devices is currently unavailable

Include any warning/errors/backtraces from the system logs

With all_syslog.sh enabled for zed, there are no dmesg entries, and no new zed output upon running the zpool import commands.

I did try zpool import -f --rewind-to-checkpoint poolname (which failed) before the above---could that have contributed to the problem?

[ahrens]

zpool import --rewind-to-checkpoint poolname is the correct way to import the checkpoint, not zpool import -T. What's printed when it fails? What's in /proc/spl/kstat/zfs/dbgmsg?

It sounds like from within the VM you can run zdb -l which is a good sign. What happens if you try to open the pool with zdb? e.g. zdb -e poolname?

[aerusso]

zpool import --rewind-to-checkpoint poolname is the correct way to import the checkpoint, not zpool import -T. What's printed when it fails?
I tried only -f --rewind-to-checkpoint first, and got "pool corrupted, restore from backup".

Are you saying that I should have been able to zpool checkpoint, just dump the constantly-changing version of the partition (it's imported, filesystems are mounted, the system is live and being written to, etc.), and then simply import it with zpool import --rewind-to-checkpoint? It would at a minimum require require -f, because the pool would still think it's imported on another system.

What's in /proc/spl/kstat/zfs/dbgmsg?

It sounds like from within the VM you can run zdb -l which is a good sign. What happens if you try to open the pool with zdb? e.g zdb -e poolname??

Unfortunately, I had to dump that 1TB image because I don't have unlimited hard drive space. But, if it would be useful for the project, I should be able to reproduce those errors at a later point. Right now though, I have to limit my bug-squashing to #11688, since that is blocking a kernel upgrade for me and I cannot descend any further into this particular stack.

[ahrens]

Are you saying that I should have been able to zpool checkpoint, just dump the constantly-changing version of the partition (it's imported, filesystems are mounted, the system is live and being written to, etc.), and then simply import it with zpool import --rewind-to-checkpoint? It would at a minimum require require -f, because the pool would still think it's imported on another system.

You're right, -f might also be needed (unless you're using -o readonly=on).