optimize recv_fix_encryption_hierarchy()

[gamanakis]

Motivation and Context

Motivation: #16317
Closes: #16317

Description

recv_fix_encryption_hierarchy() in its present state goes through all
stream filesystems, and for each one traverses the snapshots in order to
find one that exists locally. This happens by calling guid_to_name() for
each snapshot, which iterates through all children of the filesystem.
This results in CPU utilization of 100% for several minutes (for ~1000
filesystems on a Ryzen 4350G) for 1 thread at the end of a raw receive
(-w, regardless whether encrypted or not, dryrun or not).

Fix this by following a different logic: using the top_fs name, call
gather_nvlist() to gather the nvlists for all local filesystems. For
each one filesystem, go through the snapshots to find the corresponding
stream's filesystem (since we know the snapshots guid and can search
with it in stream_avl for the stream's fs). Then go on to fix the
encryption roots and locations as in its present state.

Avoiding guid_to_name() iteratively makes
recv_fix_encryption_hierarchy() significantly faster (from several
minutes to seconds for ~1000 filesystems on a Ryzen 4350G).

Another problem is the following: in case we have promoted a clone of
the filesystem outside the top filesystem specified in zfs send, zfs
receive does not fail but returns an error:
recv_incremental_replication() fails to find its origin and errors out
with needagain=1. This results in recv_fix_hierarchy() not being called
which may render some children of the top fs not mountable since their
encryption root was not updated. To circumvent this make
recv_incremental_replication() silently ignore this error.

How Has This Been Tested?

An existing test was expanded.
Perfomance testing was done locally using the script from #16317

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[gamanakis]

c63079c: rebased to master
a55562a: rebased to master

[gamanakis]

Just converted this to a draft again. The original test was broken, in the sense that it is not checking the properties on the receiving pool, but rather in the source.

The problem is that even in its current form (ie before this PR) the corrected test fails: recv_fix_encryption() fails to change the encryption root of test/fs/child after the second recv, resulting in an unmountable fs as we have already changed its key.

EDIT:
This happens because recv_incremental_replication() detects we have the wrong origin (in case we had promoted earlier a clone) and terminates with softerr=1, which leads to recv_fix_encryption_hierarchy() being skipped.

Now all this happens only when we send a fs but have previously promoted a top clone outside of the top fs used in zfs send:

zpool export -a
zpool create test /root/A.vdev -f
zpool create testB /root/B.vdev -f

echo asdfgasdfg | zfs create -o encryption=on -o keyformat=passphrase test/fs
zfs snap -r test/fs@snap
zfs clone test/fs@snap test/clone
zfs create test/fs/child

zfs snap -r test/fs@before
zfs send -Rw test/fs@before | zfs recv -dF testB

echo "****** 1st SEND Completed ******"
echo qwertqwert | zfs change-key -o keyformat=passphrase test/fs/child
echo "****** Changed Key on Child ******"
zfs promote test/clone ########### Here we are promoting outside of the top fs ##########
zfs snap -r test/fs@after
zfs send -Rwi test/fs@before test/fs@after | zfs recv -vdF testB

zfs get encryptionroot test/fs/child
zfs get encryptionroot testB/fs/child

Address this by ignoring the error, allowing recv_fix_encryption_hierarchy() to run.

[behlendorf]

This looks right to me. Thanks for running this down and implementing some more efficient logic for this.