DAOS-16927 pool: Fix upgrade DER_NOTSUPPORTEDs

[liw]

If a pool upgrade encounters an error in pool_upgrade_props, the upgrade global version remains at the old global version, while the upgrade status changes to FAILED. In this state, a repeated pool upgrade will fail with the following error in the PS leader engine log:

ds_pool_upgrade_if_needed() b377db55: upgrading pool 3 -> 4 is
  unsupported because pool upgraded to 3 last time failed

This patch implements a quick fix that ensures the upgrade global version is updated to the new global version when the upgrade status changes to FAILED, so that a repeated upgrade command will attempt to upgrade the pool again.

Before requesting gatekeeper:

• Two review approvals and any prior change requests have been resolved.
• Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
• Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
• Commit messages follows the guidelines outlined here.
• Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

• You are the appropriate gatekeeper to be landing the patch.
• The PR has 2 reviews by people familiar with the code, including appropriate owners.
• Githooks were used. If not, request that user install them and check copyright dates.
• Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
• All builds have passed. Check non-required builds for any new compiler warnings.
• Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
• If applicable, the PR has addressed any potential version compatibility issues.
• Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
• Extra checks if forced landing is requested
  • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
  • No new NLT or valgrind warnings. Check the classic view.
  • Quick-build or Quick-functional is not used.
• Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

[github-actions]

Ticket title is 'Special pool and container handles should be stored persistently in RDBs'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-16927

[liw]

Due to extremely long CI response times, I'm requesting reviews before CI results come in. The PR has passed my manual upgrade testing though.

[wangshilong]

Thanks for fixing issue.

[liw]

Thanks for fixing issue.

@wangshilong, thank you for the guidance.

[kccain]

No change requested. Question (I must have forgotten how interop/upgrade is supposed to work) - is there logic anywhere that enforces an upgrade of one layout version at a time? For example, before an upgrade attempt, if we have a pool with global_version=2, upgrade_version=2, and the software installation has DAOS_POOL_VERSION=4 - does the upgrade code just try to upgrade the pool to the latest DAOS_POOL_VERSION(4), or is some logic in place to force an upgrade to global version 2 first, then another upgrade to global version 3?