How to guarantee that a phase belongs to a the current generation?

[WillSewell]

We want to track the progress of a rollout. We do this by first polling the Rollout.status.workloadGeneration field until it matches the generation returned when the pod spec was updated.

Then we poll the Rollout.status.phase until it is either Degraded/Paused/Healthy (i.e. no longer Progressing).

The problem is, the update of status.workloadGeneration and the update of status.phase isn't atomic in all cases. So it's possible that we read the new status.workloadGeneration but the status.phase from the previous generation.

Example controller logs where it first updates status.workloadGeneration and then updates status.phase:

time="2023-07-07T09:27:10Z" level=info msg="Patched: {\"status\":{\"abort\":null,\"abortedAt\":null,\"currentPodHash\":\"674487bbb9\",\"observedGeneration\":\"3199\",\"workloadObservedGeneration\":\"1812\"}}" generation=3199 namespace=example re
sourceVersion=13927768850 rollout=example
time="2023-07-07T09:27:10Z" level=info msg="Patched conditions: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2023-07-05T15:22:21Z\",\"lastUpdateTime\":\"2023-07-05T15:22:21Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPause
d\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2023-07-07T09:04:24Z\",\"lastUpdateTime\":\"2023-07-07T09:04:24Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed
\"},{\"lastTransitionTime\":\"2023-07-07T09:09:57Z\",\"lastUpdateTime\":\"2023-07-07T09:09:57Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":
\"2023-07-07T09:27:10Z\",\"lastUpdateTime\":\"2023-07-07T09:27:10Z\",\"message\":\"Retrying Rollout after abort\",\"reason\":\"RolloutRetry\",\"status\":\"Unknown\",\"type\":\"Progressing\"}],\"message\":\"more replicas need to be updated\",\"ph
ase\":\"Progressing\"}}" generation=3199 namespace=example resourceVersion=13927768892 rollout=example

Given this I'm not sure how we can guarantee the phase we are reading is part of the desired generation. Is there some way of doing this, or an alternative way of satisfying our use case that we might not have considered? We could workaround this by adding some delay between reading the desired generation, and polling for the `status.phase, but I'd rather not if there was a less brittle alternative 🙂