How can I undo zpool offline -f?

[techhazard]

I have made a terrible mistake, but I think it might still be possible to recover from it.

In short:

• I'm running zfs 0.8.4 (with dkms) on Centos 8 (4.18.0-193.19.1.el8_2.x86_64)
• I thought I had a raidz pool (but it was just a pool of striped disks)
• I wanted to switch to a pool of mirrors so I marked one of the devices offline with zpool offline -f
• I then made the mistake of exporting the pool (by rebooting)
• Now I can't import the pool because one of the devices is "faulted" (the disk is fine, it's just the VDEV_AUX_EXTERNAL_PERSIST that has been set by zpool offline -f)
• I can't zpool online the device, because that only works on imported pools.

How can I ignore the offline flag of the device so that I can import the pool and move the data to a new pool?

For this to work I think I need either:

• a way to unset VDEV_AUX_EXTERNAL_PERSIST while the pool is exported.
• a way to ignore VDEV_AUX_EXTERNAL_PERSIST so that I can run zpool online on the device (assuming that will work)
  But there may be another solution I'm not seeing.

I already looked a bit through the codebase, but I'm no real C dev, and I'd rather not do something that will corrupt the pool any further...
These parts seem related:

• module/zfs/spa.c:2880
• module/zfs/vdev.c:718-722
• module/zfs/vdev.c:866
• module/zfs/zfs_ioctl.c:1968-1972

If I should provide more information, please let me know

[rnz]

May be help: #10254 (comment)

[techhazard]

Thanks for the suggestion!

Sadly this does not fix it.

If I do

echo 1 > /sys/module/zfs/parameters/zfs_max_missing_tvds

the action: line from zpool import changes from

The pool cannot be imported due to damaged devices or data.

to

The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported.

and the state: line changes from UVAVAIL to DEGRADED.

I can then import the pool readonly (zpool import -o readonly=on poolname), but I can't zpool clear or zpool online the disk, because both commands tell me: cannot online 13056922995524909576: pool is read-only

If I then try to zpool import poolname (without readonly=on), it tells me: cannot import 'poolname': one or more devices is currently unavailable

[rnz]

I had a similar situation - I was add big hdd to pc and sync data from read-only pool, after I destroy damaged pool.

[techhazard]

I'm currently making backups of the full disks, in the hope that I can restore this later. Thanks for the help. 🙂

[IvanVolosyuk]

This seems relevant:

zfs/module/zfs/vdev.c

Line 797 in 9305ff2

if (rc == 0 && tmp != NULL && strcmp(tmp, "external") == 0) {

I wonder if you can simply comment out this if-statement and re-compile the module.

	/*
	 * ZPOOL_CONFIG_AUX_STATE = "external" means we previously forced a
	 * fault on a vdev and want it to persist across imports (like with
	 * zpool offline -f).
	 */
	rc = nvlist_lookup_string(nv, ZPOOL_CONFIG_AUX_STATE, &tmp);
	if (rc == 0 && tmp != NULL && strcmp(tmp, "external") == 0) {
		vd->vdev_stat.vs_aux = VDEV_AUX_EXTERNAL;
		vd->vdev_faulted = 1;
		vd->vdev_label_aux = VDEV_AUX_EXTERNAL;
	}

[techhazard]

Thanks!
I'll try this after I have made some backups of the disks, which will take a few days