Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] change in behavior of unsatisfiablePDB. Unable to upgrade. #4805

Open
satyamkapoor opened this issue Feb 20, 2025 · 1 comment
Open
Assignees

Comments

@satyamkapoor
Copy link

Describe the bug
Change in behavior of upgrade error: Unsatisfiable PDB

To Reproduce
Steps to reproduce the behavior:

  1. Create a PDB with minAvailable == replicas. Ideally, place the workload on a workerpool node.
  2. Try upgrading cluster to a new k8s patch version, or just upgrading the builderpool to the new K8S patch version az aks nodepool upgrade --resource-group rgname --cluster-name cluster-name --name builderpool --no-wait --kubernetes-version 1.30.9

Expected behavior

  • Upgrade to nodepool gets performed without any issues, as the workload associated with PDB is deployed on another (workerpool) nodepool.
  • Workerpool upgrade begins where a new set of temporary nodes are created, tries to drain the old nodes, and if PDB is making it stuck, retries until a timeout and upgrade process fails.

Environment (please complete the following information):

  • CLI Version 2.68.0
  • Kubernetes version 1.30.7

Additional context

Cluster patch upgrade fails with UnsatisfiablePDB

(UnsatisfiablePDB) Upgrade is blocked due to invalid Pod Disruption Budgets (PDBs). Please review the PDB spec to allow disruptions during upgrades. To bypass this error, set forceUpgrade in upgradeSettings.overrideSettings. Bypassing this error without updating the PDB may result in drain failures during upgrade process. Invalid PDBs details: 2 errors occurred:
	* PDB ns/postgres-db-primary has minAvailable(1) >= expectedPods(1)  can't proceed with put operation
	* PDB ns/postgres-db-primary has minAvailable(1) >= expectedPods(1)  can't proceed with put operation

We're using CNPG operator, which places certain PDB on primary to avoid disruption. During an upgrade, labels on Postgres pods change to adapt the new primary, so the upgrade can complete. Unfortunately now, we see that the upgrade does not even begin due to unsatisfiablePDB. As per AKS documentation, unsatisfiablePDB is when maxUnavailable = 0. In our case minAvailable is 1, allowed disruptions are 1, and maxUnavailable is N/A.

We've had the same PDB for a year and never had issues with upgrades, except from last week. Any help is appreciated. I feel that UnsatisfiablePDB's approach is changed without updating the documentation.

Copy link
Contributor

@kaarthis, @sdesai345 would you be able to assist?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants