-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Not handled case when leader pod have status reason "Terminated" #2
Comments
Same here. I think this case could impact the availability. A new pod waits until a lock file of ConfigMap named Currently, we have only two tricky options when the node shutdown happens:
|
I think we have two approaches to resolve this issue:
Approach 1 is easier, but keeping up with k8s is harder. Approach 2 might be laborious, but it can address the root cause and be line with the de-facto standard upstream. |
In this issue, adding - podEvicted := pod.Status.Reason == "Evicted" || pod.Status.Reason == "Shutdown"
+ podEvicted := pod.Status.Reason == "Evicted" || pod.Status.Reason == "Terminated" |
I think best solution will be without check reason, just switching leader and delete failed pod. |
I have a case when all nodes of k8s cluster was rebooted, and pod with operator have a status
Failed
and reasonTerminated
.Library does not handle this case, and Vitess cluster not start after accident on k8s cluster. New pod wait when old leader will be deleted and do nothing.
The text was updated successfully, but these errors were encountered: