Improve e2e test framework to enable better checking for brief conditions

[juliev0]

Summary

There are some things our e2e test can't effectively test without false failures. The problem is that sometimes the state of things is brief enough that an Eventually call will fail.

If we have another goroutine watching/storing the resource changes, then we could theoretically have a test which can check if a particular condition happened in a given period of time (needs more design).

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.