wait_until_pods_running should be able to rule out error pods that are not related to the final state of the owners

[chizhg]

In

test-infra/scripts/library.sh

Line 133 in 97901db

function wait_until_pods_running() {

, wait_until_pods_running will only succeed if all pods in the given namespace are in Running or Completed state.

But since k8s has some retry logic, e.g. K8s Job can create a new pod if there is an error, one error pod does not necessarily mean the Job fails - https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/knative_serving/6440/pull-knative-serving-integration-tests/1214219978266382337 is an example. In such scenario wait_until_pods_running will return an error that is not necessarily true.

This function should be general enough to consider and rule out error pods that are not related to the final state of the owners, e.g.

1. For Deployments, all pods should be X/X Running
2. For Jobs, it should depend on the success criteria
   ...

FYI @mattmoor

[adrcunha]

Please clarify what are the specific requirements for undoubtedly ruling a collection of pods as "running". If the answer is "the requirements depend on what we're waiting to be running", then this function should be removed from library.sh and the required variants implemented in each place it's used.

[steuhs]

Alternatively we can delete those failed ones after retry started, we don't need to make the change here

[chizhg]

We are going to reimplement this function with Go and won't make any incremental changes to this function.

/remove-kind good-first-issue
/lifecycle frozen

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. \nReopen the issue with /reopen.\nMark the issue as
fresh by adding the comment /remove-lifecycle stale.

[chizhg]

/reopen

[mattmoor]

Seems to me like this entire function could just be:

kubectl wait pod --for=condition=Ready -n $1 -l '!job-name'

This doesn't properly check jobs, but what we have today is fairly hit or miss.

[chizhg]

Thanks @mattmoor ! I have created #2440

[chizhg]

#2440 will be partially reverted by #2443, so reopen this issue.