From 0a8bcfb6b6f59f4b7ddc67063285a11f7ba839db Mon Sep 17 00:00:00 2001 From: Flavio Castelli Date: Wed, 5 Mar 2025 18:32:26 +0100 Subject: [PATCH] doc(mtls): security hardening of the webhooks Explain how to secure the webhooks provided by Kubewarden by using mTLS and Kubernetes Network policies. Moreover, update the ArgoCD documentation to include the extra operations to perform when mTLS is enabled. Signed-off-by: Flavio Castelli --- docs/howtos/argocd-installation.md | 6 + .../howtos/security-hardening/_category_.json | 5 + .../howtos/security-hardening/webhook-mtls.md | 159 ++++++++++++ .../security-hardening/_category_.json | 5 + .../security-hardening/webhook-mtls.md | 244 ++++++++++++++++++ docs/reference/threat-model.md | 3 + 6 files changed, 422 insertions(+) create mode 100644 docs/howtos/security-hardening/_category_.json create mode 100644 docs/howtos/security-hardening/webhook-mtls.md create mode 100644 docs/reference/security-hardening/_category_.json create mode 100644 docs/reference/security-hardening/webhook-mtls.md diff --git a/docs/howtos/argocd-installation.md b/docs/howtos/argocd-installation.md index 554233f9d22..222cdebac38 100644 --- a/docs/howtos/argocd-installation.md +++ b/docs/howtos/argocd-installation.md @@ -94,6 +94,12 @@ spec: namespace: kubewarden-system jsonPointers: - /data + - group: "" + kind: "Secret" + name: kubewarden-audit-scanner-client-cert + namespace: kubewarden-system + jsonPointers: + - /data - group: "admissionregistration.k8s.io" kind: "MutatingWebhookConfiguration" jqPathExpressions: diff --git a/docs/howtos/security-hardening/_category_.json b/docs/howtos/security-hardening/_category_.json new file mode 100644 index 00000000000..b2ea062b3a1 --- /dev/null +++ b/docs/howtos/security-hardening/_category_.json @@ -0,0 +1,5 @@ +{ + "label": "Security", + "position": 100, + "collapsed": true +} diff --git a/docs/howtos/security-hardening/webhook-mtls.md b/docs/howtos/security-hardening/webhook-mtls.md new file mode 100644 index 00000000000..c9b0b6ef001 --- /dev/null +++ b/docs/howtos/security-hardening/webhook-mtls.md @@ -0,0 +1,159 @@ +--- +sidebar_label: Enable mTLS with k3s +title: Secure webhooks with mutual TLS with k3s +description: Harden the webhook configuration. +keywords: [kubewarden, kubernetes, security] +doc-persona: [kubewarden-operator, kubewarden-integrator] +doc-type: [howto] +doc-topic: [operator-manual, security] +--- + + + + + +This guide shows you how to enable mutual TLS (mTLS) for all the webhooks used by the Kubewarden +stack when using [k3s](https://k3s.io/) as your Kubernetes distribution. + +## Prerequisites + +Before installing k3s, you need to create a certificate authority (CA) and a client certificate to use to secure the communication between the Kubewarden webhooks and the Kubernetes API server. + +As a first step, create the `/etc/rancher/k3s/admission/certs` directory: + +```console +sudo mkdir -p /etc/rancher/k3s/admission/certs +``` + +### Create a root CA and the client certificate + +As `root` user, change directory to the `/etc/rancher/k3s/admission/certs` directory and +create all needed certificates: + +```console +export FQDN=mtls.kubewarden.io + +# Create CA +openssl req -nodes -batch -x509 -sha256 -days 365 -newkey rsa:2048 -keyout rootCA.key -out rootCA.crt + +# Create CSR +openssl req -nodes -batch -newkey rsa:2048 -keyout client.key -out client.csr \ + -addext "subjectAltName = DNS:$FQDN" + +# Create CRT +openssl x509 -req -CA rootCA.crt -CAkey rootCA.key -in client.csr -out client.crt -days 365 -CAcreateserial \ + -extfile <(echo "subjectAltName=DNS:$FQDN") + +# Print CRT +openssl x509 -text -noout -in client.crt +``` + +The following files should have been created: + +- `client.crt` +- `client.csr` +- `client.key` +- `rootCA.crt` +- `rootCA.key` +- `rootCA.srl` + +### Create the Kubernetes configuration file + +Create the `/etc/rancher/admission/admission.yaml` file with the following content: + +```yaml +# /etc/rancher/admission/admission.yaml +apiVersion: apiserver.config.k8s.io/v1 +kind: AdmissionConfiguration +plugins: +- name: ValidatingAdmissionWebhook + configuration: + apiVersion: apiserver.config.k8s.io/v1 + kind: WebhookAdmissionConfiguration + kubeConfigFile: "/etc/rancher/k3s/admission/kubeconfig" +- name: MutatingAdmissionWebhook + configuration: + apiVersion: apiserver.config.k8s.io/v1 + kind: WebhookAdmissionConfiguration + kubeConfigFile: "/etc/rancher/k3s/admission/kubeconfig" +``` + +Finally, create a `kubeconfig` file at `/etc/rancher/k3s/admission/kubeconfig`: + +```yaml +# /etc/rancher/admission/kubeconfig +apiVersion: v1 +kind: Config +users: +- name: '*.kubewarden.svc' # namespace where the kubewarden stack is deployed + user: + client-certificate: /etc/rancher/k3s/admission/certs/client.crt + client-key: /etc/rancher/k3s/admission/certs/client.key +``` + +### Create a k3s configuration file + +Create a k3s configuration file at `/etc/rancher/k3s/config.yaml`: + +```yaml +# /etc/rancher/k3s/config.yaml +kube-apiserver-arg: +- admission-control-config-file=/etc/rancher/k3s/admission/admission.yaml +``` + +## Install k3s + +Install k3s using the following command: + +```console +curl -sfL https://get.k3s.io | sh - +``` + +Wait for the installation to complete. + +## Install the Kubewarden stack + +### Prerequisites + +The certificate of the root CA, that issued the Kubernetes client certificate, must be made available to +the Kubewarden stack. + +The root CA is available at `/etc/rancher/k3s/admission/certs/rootCA.crt` on the Kubernetes node. Its content +has to be put into a `ConfigMap` under the `kubewarden` namespace. The contents of the `rootCA.crt` file +must be stored in a key named `client-ca.crt`. + +First, create the `kubewarden` namespace: + +```console +kubectl create namespace kubewarden +``` + +Then create the `ConfigMap` in it. The following command, run on the Kubernetes node, +does that: + +```console +kubectl create configmap -n kubewarden api-server-mtls \ + --from-file=client-ca.crt=/etc/rancher/k3s/admission/certs/rootCA.crt +``` + +The resulting `ConfigMap` is named `api-server-mtls`. + +### Install the Kubewarden stack + +Install the Kubewarden stack as described in the [quickstart guide](../../quick-start.md). +Follow all the steps, but when installing the `kubewarden-controller` Helm chart, make sure to +enable the following values: + +- `mTLS.enable`: must be set to `true`. +- `mTLS.configMapName`: must be set to name of the `ConfigMap` that was previously created. + +Given the `ConfigMap` was named `api-server-mtls`, the Helm command to install the `kubewarden-controller` +is: + +```console +helm install --wait -n kubewarden kubewarden-controller kubewarden/kubewarden-controller \ + --set mTLS.enable=true \ + --set mTLS.configMapName=api-server-mtls +``` + +Once this command finishes, the Kubewarden stack is installed and its webhooks are secured with mTLS. diff --git a/docs/reference/security-hardening/_category_.json b/docs/reference/security-hardening/_category_.json new file mode 100644 index 00000000000..550acd130ba --- /dev/null +++ b/docs/reference/security-hardening/_category_.json @@ -0,0 +1,5 @@ +{ + "label": "Security", + "position": 70, + "collapsed": true +} diff --git a/docs/reference/security-hardening/webhook-mtls.md b/docs/reference/security-hardening/webhook-mtls.md new file mode 100644 index 00000000000..7526ed1e1fd --- /dev/null +++ b/docs/reference/security-hardening/webhook-mtls.md @@ -0,0 +1,244 @@ +--- +sidebar_label: Webhook Mutual TLS +title: Secure webhooks with mutual TLS +description: Harden the webhook configuration. +keywords: [kubewarden, kubernetes, security] +doc-persona: [kubewarden-operator, kubewarden-integrator] +doc-type: [reference] +doc-topic: [reference, security] +--- + + + + + +The Kubewarden stack uses various webhooks to enforce policies in a Kubernetes cluster. Each `PolicyServer` instance exposes a webhook +that the Kubernetes API server calls to validate and mutate resources. Moreover, the `kubewarden-controller` exposes webhooks to validate and mutate +the custom resources provided by the Kubewarden project. + +To decrease their attack surface, you should limit access to these webhooks to the only valid callers they have: + +- the Kubernetes API server +- the [audit scanner](../../explanations/audit-scanner/) component. +You can do this using network policies and authentication independently, or together, to harden the webhooks against attacks. + +## Block External Traffic Using Network Policies + +Webhooks are only expected to accept requests from the Kubernetes API server and the audit scanner component. By default, however, webhooks can accept traffic from any source. +If you are using a CNI that supports Network Policies, you can create a policy that blocks traffic that doesn't originate from the API server. + +The built-in NetworkPolicy resource in Kubernetes can't block or admit traffic from the cluster hosts, and the `kube-apiserver` process is always running on the host network. +Therefore, you must use the advanced network policy resources from the CNI in use. Examples for Calico and Cilium follow. Consult the documentation for your CNI for more details. + +### Calico + +Use the NetworkPolicy resource in the `crd.projectcalico.org/v1` API group, define a network policy like the following one: + +```yaml +apiVersion: crd.projectcalico.org/v1 +kind: NetworkPolicy +metadata: + name: allow-k8s-and-audit-scanner + namespace: kubewarden # namespace where the kubewarden stack is deployed +spec: + selector: 'app.kubernetes.io/component in {"kubewarden-controller", "policy-server"}' + types: + - Ingress + ingress: + # this allows the Kubernetes API server to reach the kubewarden controller and + # all the policy server instances + - action: Allow + protocol: TCP + source: + nets: + # CIDR of the control plane host. May list more than 1 if the hosts are in different subnets. + - 192.168.42.0/24 + destination: + selector: 'app.kubernetes.io/component in {"kubewarden-controller", "policy-server"}' + # this allows all the workloads defined inside of the kubewarden namespace + # (including audit-scanner) to reach the kubewarden controller and all the + # policy server instances + - action: Allow + protocol: TCP + source: + # namespace where the kubewarden stack is deployed + namespaceSelector: 'kubernetes.io/metadata.name == "kubewarden"' # namespace where the kubewarden stack is deployed + destination: + selector: 'app.kubernetes.io/component in {"kubewarden-controller", "policy-server"}' +``` + +:::warning + +This network policy uses label selectors that have been introduced in Kubewarden 1.23.0. If you are using an older version, +you must update the labels in the policy to match the ones used in your deployment. + +More specifically, the + +```yaml +selector: 'app.kubernetes.io/component in {"kubewarden-controller", "policy-server"}' +``` + +selectors should be written as: + + +```yaml +selector: 'app.kubernetes.io/name == "kubewarden-controller" || has(kubewarden/policy-server)' +``` + +::: + +### Cilium + +Use the CiliumNetworkPolicy resource in the `cilium.io/v2` API group to define a network policy like the following one: + +```yaml +apiVersion: "cilium.io/v2" +kind: CiliumNetworkPolicy +metadata: + name: allow-k8s-and-audit-scanner + namespace: kubewarden # namespace where the kubewarden stack is deployed +spec: + endpointSelector: + matchExpressions: + - key: app.kubernetes.io/component + operator: In + values: + - policy-server + - controller + ingress: + # this allows the Kubernetes API server to reach the kubewarden controller and + # all the policy server instances + - fromEntities: + - host + - remote-node + # this allows all the workloads defined inside of the kubewarden namespace + # (including audit-scanner) to reach the kubewarden controller and all the + # policy server instances + - fromEndpoints: + - matchLabels: + # namespace where the kubewarden stack is deployed + k8s:io.kubernetes.pod.namespace: kubewarden +``` + +:::warning + +This network policy uses label selectors that have been introduced in Kubewarden 1.23.0. If you are using an older version, +you must update the labels in the policy to match the ones used in your deployment. + +More specifically, the + +```yaml +matchExpressions: + - key: app.kubernetes.io/component + operator: In + values: + - policy-server + - controller +``` + +expression should be written as: + + +```yaml +endpointSelector: +matchExpressions: + - key: app.kubernetes.io/name + operator: In + values: + - kubewarden-controller + - key: kubewarden/policy-server + operator: Exists +``` + +::: + +## Require the Kubernetes API Server to Authenticate to the Webhook + +:::tip + +See [this how-to](../../howtos/security-hardening/webhook-mtls/) for a step-by-step guide on +configuring the Kubernetes API server of k3s to authenticate to the webhook. + +::: + +The webhooks exposed by the Kubewarden stack should only accept requests from the Kubernetes API server or +from the audit scanner component. +By default, these webhooks don't require clients to authenticate to it. They will accept any request. + +You can configure the webhooks to require credentials so that only the API server and the audit scanner processes +can access them. +See the [Kubernetes documentation](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#authenticate-apiservers) for more information. + +1. Configure the API server to present a client certificate to the webhook, pointing to an `AdmissionConfiguration` + file to configure the `ValidatingAdmissionWebhook` and `MutatingAdmissionWebhook` plugins: + + Create a file named `admission.yaml` with the following contents: + + ```yaml + apiVersion: apiserver.config.k8s.io/v1 + kind: AdmissionConfiguration + plugins: + - name: ValidatingAdmissionWebhook + configuration: + apiVersion: apiserver.config.k8s.io/v1 + kind: WebhookAdmissionConfiguration + kubeConfigFile: "/etc/k8s/admission/kubeconfig" + - name: MutatingAdmissionWebhook + configuration: + apiVersion: apiserver.config.k8s.io/v1 + kind: WebhookAdmissionConfiguration + kubeConfigFile: "/etc/k8s/admission/kubeconfig" + ``` + + This is the same configuration file used to configure other plugins, such as `PodSecurity`. + If your distribution or setup uses additional admission plugins, you should also configure those. + +2. Create the `kubeconfig` file the admission plugins refer to. Kubewarden only supports client certificate authentication, so generate a TLS key pair, + and set the kubeconfig to use either client-certificate and client-key or client-certificate-data and client-key-data. + + For example: + + ```yaml + # /etc/k8s/admission/kubeconfig + apiVersion: v1 + kind: Config + users: + - name: '*.kubewarden.svc' + user: + client-certificate: /path/to/client/cert + client-key: /path/to/client/key + ``` + +3. Start the `kube-apiserver` binary with the flag `--admission-control-config-file` pointing to your `AdmissionConfiguration` file. + The way to do this varies by distribution, and it isn't supported universally, such as in hosted Kubernetes providers. + Consult the documentation for your Kubernetes distribution. + +4. The certificate of the root CA that issued the API server client certificate must be made available to + the Kubewarden stack. + + Its content has to be put into a `ConfigMap` under the `kubewarden` namespace using a key named `client-ca.crt`. + + Assuming the root CA is available at `/etc/k8s/admission/certs/rootCA.crt`, create the `ConfigMap` + with the following command: + + ```console + kubectl create configmap -n kubewarden api-server-mtls \ + --from-file=client-ca.crt=/etc/k8s/admission/certs/rootCA.crt + ``` + +5. Finally, when installing the `kubewarden-controller` Helm chart, + make sure to enable the following values: + + - `mTLS.enable`: must be set to `true`. + - `mTLS.configMapName`: must be set to name of the `ConfigMap` that was previously created. + + Given the `ConfigMap` was named `api-server-mtls`, the Helm command to install the `kubewarden-controller` is: + + ```console + helm install --wait -n kubewarden kubewarden-controller kubewarden/kubewarden-controller \ + --set mTLS.enable=true \ + --set mTLS.configMapName=api-server-mtls + ``` + + The Kubewarden controller creates a client certificate for use by the audit scanner component. + The certificate is automatically rotated, by the controller, when needed. diff --git a/docs/reference/threat-model.md b/docs/reference/threat-model.md index f28fc7f11d0..242310f5257 100644 --- a/docs/reference/threat-model.md +++ b/docs/reference/threat-model.md @@ -135,6 +135,8 @@ and admission controller webhook. Configure the cluster with mTLS authentication for the Webhooks and enable the mTLS feature in the Kubewarden stack. Alternatively, setup mTLS using a CNI that supports Network Policies. +See [here](./security-hardening/webhook-mtls.md) for more information. + Use the [capabilities-psp](https://artifacthub.io/packages/kubewarden/capabilities-psp/capabilities-psp) policy and configure it to drop NET_RAW capabilities. @@ -150,6 +152,7 @@ for the admission controller webhook, by spoofing. Configure the cluster with mTLS authentication for the Webhooks and enable the mTLS feature in the Kubewarden stack. Alternatively, setup mTLS using a CNI that supports Network Policies. +See [here](./security-hardening/webhook-mtls.md) for more information. ### Threat 10 - Abusing a mutation rule to create a privileged container