Help determining the source of "dataset is busy" when running "zfs destroy"

[amartin3225]

Hello,

Running different versions of OpenZFS on Ubuntu 20.04 (e.g. 0.8.3 or 2.1.6) with LXD, I frequently encounter the following error when attempting to delete a dataset even if the LXD container is stopped and nothing is using the dataset:

# zfs destroy -r lxd/containers/mycontainer
cannot destroy 'lxd/containers/mycontainer': dataset is busy

As far as I can tell, NOTHING is using this dataset:

• mountpoint is set to legacy and lsof returns no processes related to it
• lxd/containers/mycontainer/%recv does not exist
• the dataset is not listed in /proc/*/mounts
• an NFS server is not installed on this system and sharenfs is set to off on this dataset
• there are no snapshots, bookmarks, or holds
• there are no zfs send, zfs receive, etc processes running
• all LXD processes are stopped
• /proc/spl/kstat/zfs/dbgmsg doesn't show anything when running the above command

If I try to strace the above command, I see the following:

openat(AT_FDCWD, "/dev/zfs", O_RDWR|O_CLOEXEC) =3D 5
...
ioctl(5, _IOC(_IOC_NONE, 0x5a, 0x18, 0), 0x7ffd7a1de290) =3D -1 EBUSY (Device or resource busy)

Moreover, I am unable to export this pool:

# zpool export -f lxd
cannot export 'lxd': pool is busy

How can I determine what is causing this dataset to remain "busy"? Is there a way for me to use bpftrace (please help with the bpftrace command I would need to use) to see what ZFS sees as holding open this dataset? A reboot will fix it, but I cannot frequently reboot this production system.

Thanks!

[amartin3225]

Additional work with bpftrace indicates that the EBUSY is coming from dsl_destroy_head_check_impl() but I have been unsuccessful in attempting to compile a custom version of ZFS with printks throughout this function to learn more.

[jeffrey-aguilera]

I have a similar issue. No clue why zfs thinks everything is busy, when nothing is.

[rincebrain]

AFAIK ZFS has no cases where it's incorrect about that, the problem is that there's nothing but a simple refcount without an explanation of who's holding it for those checks.

In general, the 4 most common reasons are:

• NFS exports don't show up in lsof and will hold things open
• someone did umount -l and there's a lingering reference even though you don't see it in mounts
• mount namespacing can mean it was mounted somewhere that's not easily visible in /proc/mounts and then umount -l happened
• having older versions of kvm/qemu open used to hold open every mountpoint and zvol on the system at the time they started for some reason, i have no idea why.

(I know you listed reasons to eliminate those, I was just enumerating it for anyone running into it who reads this.)

Do you have more of a stack trace for who was calling dsl_destroy_head?

You could also set zfs_flags to 1 or 513 and try again and you'll get quite a flood of output of what returned any non-"okay" return code, in the latter case.