I researched this a bit, because I noticed this behaviour as well. This is what I found:
(quotes taken from mcuboot docs)

After a new image is flashed, the bootloader (mcuoot) is instructed to perform

BOOT_SWAP_TYPE_TEST: Boot the contents of the secondary slot by swapping images. Unless the swap is made permanent, revert back on the next boot.

The swap is made permanent by InfiniTime setting a bit in the mcuboot image trailer on user validation.
This bit, however, is not retained for the swapped out "known good" firmware.
This seems odd, but actually is a feature of mcuboot, because by setting this bit in a firmware in slow two, one can perform a directly validated replacement

BOOT_SWAP_TYPE_PERM: Permanently swap images, and boot the upgraded image firmware.

For a revert from a non validated firmware (bit was not set) mcuboot calls for the old firmware to mark it self valid again.

BOOT_SWAP_TYPE_REVERT: A previous test swap was not made permanent; swap back to the old image whose data are now in the secondary slot. If the old image marks itself “OK” when it boots, the next boot will have swap type BOOT_SWAP_TYPE_NONE.

In summary, this means it is on us to retain the information that an image was good, if we want it to be automatically validated on revert. This would not require changes to the boot loader.

I see two general possibilities:

1. Store this in the firmware itself. This would mean we would have to set aside a bit somewhere in the binary, make sure compiled code never gets there and default initialize it to "not set". That should be doable with the linker, however, I have no experience with this. If the firmware is validated, this bit is also set (in addition to mcuboots flag). From then on, it would be swapped in and out by mcuboot the same way as anything else in the code. Then the image could test for this bit on boot and validate itself automatically if indicated.
2. Store this in a file on the SPI flash. This would generally employ the same strategy as storing directly in the firmware, but just write to a file in LittleFS, which would not be touched at all by mcuboot. This would then mean that it could not just be a bit, but rather something like the commit hash of the firmware.

I think number two is the pragmatic solution and I am confident I could program this.
Number one sounds cooler though 😉

Maybe someone has a completely different idea, I will wait a while for feedback before I start writing any code to fix this.

Another implication of all this is that we could set the Image OK bit in official release binaries, making mcuboot trigger the BOOT_SWAP_TYPE_PERM instead of the BOOT_SWAP_TYPE_TEST that is now used and thus remove the need to validate official, well tested releases.

I explored option 1. Out of interest and for the challenge.
I was able to use the linker to set aside memory to save the validation information in and everything compiled and worked great, but the bootloader would never revert to the image with my changes.
It turns out there is a signature that protects the image integrity and changing a bit in the image makes the signature validation fail 🤦. There would be a way around this:

For low performance MCU’s where the validation is a heavy process at boot (~1-2 seconds on a arm-cortex-M0), the MCUBOOT_VALIDATE_PRIMARY_SLOT_ONCE could be used. This option will cache the validation result as described above into the magic area of the primary slot. The next boot, the validation will be skipped if the previous validation was successfull. This option is reducing the security level since if an attacker could modify the contents of the flash after a good image has been validated, the attacker could run his own image without running validation again. Enabling this option should be done with care.

But I think there is no need to enable this when there still is option 2., so option 1. is dead for now.

I implemented option 2. It works, I have tested it multiple times on hardware. PR is #2225.