Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kill-host-pods.py: filter pods by node #4819

Merged
merged 1 commit into from
Jan 14, 2025
Merged

Conversation

petrutlucian94
Copy link
Contributor

@petrutlucian94 petrutlucian94 commented Jan 10, 2025

Starting with k8s 1.32, AuthorizeNodeWithSelectors is enabled by default: https://kubernetes.io/docs/reference/access-authn-authz/node/

If the rbac microk8s addon is enabled, the kube-apiserver will run with "--authorization-mode=RBAC,Node". This means that kublets (system:node:$node) will no longer be allowed to access pods that reside on other nodes.

For this reason, the "kill-host-pods.py" script is now getting access denied errors:

  Error from server (Forbidden): pods is forbidden:
  User "system:node:myhostname" cannot list resource "pods" in API group ""
  at the cluster scope: can only list/watch pods with spec.nodeName field selector

As suggested by the error message, we'll solve it by filtering pods by the node name.

Fixes: #4802

Summary

Changes

Testing

Possible Regressions

Checklist

  • Read the contributions page.
  • Submitted the CLA form, if you are a first time contributor.
  • The introduced changes are covered by unit and/or integration tests.

Notes

Starting with k8s 1.32, AuthorizeNodeWithSelectors is enabled by
default:

https://kubernetes.io/docs/reference/access-authn-authz/node/

If the rbac microk8s addon is enabled, the kube-apiserver will
run with "--authorization-mode=RBAC,Node". This means that
kublets (system:node:$node) will no longer be allowed to access
pods that reside on other nodes.

For this reason, the "kill-host-pods.py" script is now getting
access denied errors:

  Error from server (Forbidden): pods is forbidden:
  User "system:node:myhostname" cannot list resource "pods" in API group ""
  at the cluster scope: can only list/watch pods with spec.nodeName field selector

As suggested by the error message, we'll solve it by filtering
pods by the node name.

Fixes: #4802
Copy link
Member

@berkayoz berkayoz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, I'll run some manual tests and merge if all is well

@petrutlucian94
Copy link
Contributor Author

LGTM overall, I'll run some manual tests and merge if all is well

Thanks! fwiw, I used the following to trigger a pod cleanup:

sudo touch /var/snap/microk8s/current/var/lock/snapdata-mounts-need-reload

Copy link
Member

@berkayoz berkayoz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@berkayoz berkayoz merged commit 45a6abb into master Jan 14, 2025
12 checks passed
@berkayoz berkayoz deleted the lpetrut/fix-pods-restart branch January 14, 2025 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

kill-host-pods permission error after upgrade to 1.32
2 participants