Tenant users can patch the kube-system namespace in the latest version v0.7.2

[gowthamvasan]

Bug description

Tenant users and owners can patch the kube-system namespace alone. When tried to patch the other namespaces , the request is denied.
I know from v0.7.1 the CVE-2024-39690 issue is addressed. But this is happening only with kube-system namespace. we tested with different AKS version and results are same.

How to reproduce

Steps to reproduce the behavior:

1. Provide the Capsule Tenant YAML definitions
2. Provide all managed Kubernetes resources

Expected behavior

Tenant users should not be able to modify labels or patch other namespaces

Logs

***** below are the logs of denied namespaces ******
{"level":"debug","ts":"2024-12-18T14:03:12.159Z","logger":"events","msg":"Namespace testcapsule01 can not be patched","type":"Warning","object":{"kind":"Namespace","name":"testcapsule01","uid":"46a448e7-12f8-4420-ac10-affde5ba5c78","apiVersion":"v1","resourceVersion":"13936"},"reason":"OfflimitNamespace"}
{"level":"debug","ts":"2024-12-18T14:03:12.518Z","logger":"events","msg":"Namespace testcapsule01 can not be patched","type":"Warning","object":{"kind":"Namespace","name":"testcapsule01","uid":"46a448e7-12f8-4420-ac10-affde5ba5c78","apiVersion":"v1","resourceVersion":"13936"},"reason":"OfflimitNamespace"}
{"level":"debug","ts":"2024-12-18T14:03:22.315Z","logger":"events","msg":"Namespace capsule-system can not be patched","type":"Warning","object":{"kind":"Namespace","name":"capsule-system","uid":"90387be3-1b47-4212-9475-443fc8b8171f","apiVersion":"v1","resourceVersion":"10344"},"reason":"OfflimitNamespace"}
{"level":"debug","ts":"2024-12-18T14:03:22.660Z","logger":"events","msg":"Namespace capsule-system can not be patched","type":"Warning","object":{"kind":"Namespace","name":"capsule-system","uid":"90387be3-1b47-4212-9475-443fc8b8171f","apiVersion":"v1","resourceVersion":"10344"},"reason":"OfflimitNamespace"}
{"level":"debug","ts":"2024-12-18T14:03:30.807Z","logger":"events","msg":"Namespace nginx can not be patched","type":"Warning","object":{"kind":"Namespace","name":"nginx","uid":"09e7eca4-c1ee-4bc1-8ae5-e02dfd4d0994","apiVersion":"v1","resourceVersion":"13797"},"reason":"OfflimitNamespace"}
{"level":"debug","ts":"2024-12-18T14:03:31.160Z","logger":"events","msg":"Namespace nginx can not be patched","type":"Warning","object":{"kind":"Namespace","name":"nginx","uid":"09e7eca4-c1ee-4bc1-8ae5-e02dfd4d0994","apiVersion":"v1","resourceVersion":"13797"},"reason":"OfflimitNamespace"}

**** below the logs when tried to patch kube-system ********
{"level":"info","ts":"2024-12-18T14:04:37.113Z","logger":"controllers.Tenant","msg":"Ensuring limit resources count is updated","Request.Name":"blk0001700"}
{"level":"info","ts":"2024-12-18T14:04:37.113Z","logger":"controllers.Tenant","msg":"Ensuring all Namespaces are collected","Request.Name":"blk0001700"}
{"level":"info","ts":"2024-12-18T14:04:37.128Z","logger":"controllers.Tenant","msg":"Starting processing of Namespaces","Request.Name":"blk0001700","items":2}
{"level":"debug","ts":"2024-12-18T14:04:37.129Z","logger":"events","msg":"Ensuring Namespace metadata","type":"Normal","object":{"kind":"Tenant","name":"blk0001700","uid":"62cde0a4-cbcd-46d0-93fa-d26f5895aa22","apiVersion":"capsule.clastix.io/v1beta2","resourceVersion":"25152"},"reason":"blk0001700-testcap"}
{"level":"info","ts":"2024-12-18T14:04:37.148Z","logger":"controllers.Tenant","msg":"Starting processing of Network Policies","Request.Name":"blk0001700"}
{"level":"debug","ts":"2024-12-18T14:04:37.148Z","logger":"events","msg":"Ensuring Namespace metadata","type":"Normal","object":{"kind":"Tenant","name":"blk0001700","uid":"62cde0a4-cbcd-46d0-93fa-d26f5895aa22","apiVersion":"capsule.clastix.io/v1beta2","resourceVersion":"25152"},"reason":"kube-system"}
{"level":"info","ts":"2024-12-18T14:04:37.148Z","logger":"controllers.Tenant","msg":"Pruning objects with label selector capsule.clastix.io/network-policy","Request.Name":"blk0001700"}
{"level":"info","ts":"2024-12-18T14:04:37.148Z","logger":"controllers.Tenant","msg":"Pruning objects with label selector capsule.clastix.io/network-policy","Request.Name":"blk0001700"}
{"level":"info","ts":"2024-12-18T14:04:37.166Z","logger":"controllers.Tenant","msg":"Starting processing of Limit Ranges","Request.Name":"blk0001700","items":0}
{"level":"info","ts":"2024-12-18T14:04:37.166Z","logger":"controllers.Tenant","msg":"Pruning objects with label selector capsule.clastix.io/limit-range","Request.Name":"blk0001700"}
{"level":"info","ts":"2024-12-18T14:04:37.168Z","logger":"controllers.Tenant","msg":"Pruning objects with label selector capsule.clastix.io/limit-range","Request.Name":"blk0001700"}
{"level":"info","ts":"2024-12-18T14:04:37.180Z","logger":"controllers.Tenant","msg":"Starting processing of Resource Quotas","Request.Name":"blk0001700","items":0}
{"level":"info","ts":"2024-12-18T14:04:37.180Z","logger":"controllers.Tenant","msg":"Pruning objects with label selector capsule.clastix.io/resource-quota","Request.Name":"blk0001700"}
{"level":"info","ts":"2024-12-18T14:04:37.180Z","logger":"controllers.Tenant","msg":"Pruning objects with label selector capsule.clastix.io/resource-quota","Request.Name":"blk0001700"}
{"level":"info","ts":"2024-12-18T14:04:37.194Z","logger":"controllers.Tenant","msg":"Ensuring RoleBindings for Owners and Tenant","Request.Name":"blk0001700"}
{"level":"info","ts":"2024-12-18T14:04:37.194Z","logger":"controllers.Tenant","msg":"Pruning objects with label selector capsule.clastix.io/role-binding,capsule.clastix.io/role-binding notin (33e7d543b483f8f5,550bfb89c4b2ec05,5e46d2bbf3cfc35,e1faf6cd14fbcdad,edd505496b67ac5)","Request.Name":"blk0001700"}
{"level":"info","ts":"2024-12-18T14:04:37.194Z","logger":"controllers.Tenant","msg":"Pruning objects with label selector capsule.clastix.io/role-binding,capsule.clastix.io/role-binding notin (33e7d543b483f8f5,550bfb89c4b2ec05,5e46d2bbf3cfc35,e1faf6cd14fbcdad,edd505496b67ac5)","Request.Name":"blk0001700"}

Additional context

• Capsule version: v0.7.2
• Helm Chart version: capsule-0.7.2
• Kubernetes version: v1.30.1

[gowthamvasan]

Upon our investigation , we found that namespaces having labels "control-plane: 'true' 'kubernetes.azure.com/managedby: aks' " can be updated/patched by the tenant users. Is that anything we can remove from webhooks ?
@prometherion

[aslafy-z]

See Azure/AKS#4002
This is a "feature". You can disable it by labelling admission webhook configurations like in https://learn.microsoft.com/en-us/azure/aks/faq#can-admission-controller-webhooks-impact-kube-system-and-internal-aks-namespaces-.
However, you should ensure the webhook does not interfere with aks operations: upgrades, add-ons, commands, etc.

Capsule team: I couldn't find references of this in the coaks documentation. Have you seen this already?

[prometherion]

I'm sorry. I'm still moving to the new house. I'm a bit unresponsive and busy with boxes, but not a busybox.

@aslafy-z thanks a lot for spotting this, I didn't know and I'm grateful you shared this: we definitely need to address this in the CoAKS repository /cc @bsctl