Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cross-namespace KongPlugin doc #7578

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions app/_data/docs_nav_kic_3.2.x.yml
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,8 @@ items:
url: /plugins/mtls
- text: OIDC
url: /plugins/oidc
- text: Cross-namespace Plugins
url: /plugins/cross-namespace

- title: Reference
icon: /assets/images/icons/documentation/icn-references-color.svg
Expand Down
4 changes: 4 additions & 0 deletions app/_includes/md/kic/consumer.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
{% comment %}This file is intentionally indented as it's included in an <ol> on multiple pages{% endcomment %}
{% assign name = include.name | default: 'kotenok' %}
{%- assign namespace = include.namespace %}
{%- assign credName = include.credName %}
```bash
echo "apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
name: {{ name }}
{% if namespace -%}
namespace: {{ namespace }}
{% endif -%}
annotations:
kubernetes.io/ingress.class: kong
username: {{ name }}
Expand Down
3 changes: 3 additions & 0 deletions app/_includes/md/kic/prerequisites.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,9 @@
- name: proxy
port: 80
protocol: HTTP
allowedRoutes:
namespaces:
from: All
" | kubectl apply -f -
```

Expand Down
333 changes: 333 additions & 0 deletions app/_src/kubernetes-ingress-controller/plugins/cross-namespace.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,333 @@
---
title: Using plugins across namespaces
type: how-to
purpose: |
Configure plugins in one namespace that interact with resources in another.
---

Plugins can interact with resources across different Kubernetes namespaces.
_KongClusterPlugin_ resources create plugins available for resources in any
namespace, but require cluster-level permissions and may not be appropriate for
all environments. This guide covers the use of _ReferenceGrant_ resources to
grant selective permissions to allow _KongPlugin_ resources in one namespace to
modify resources in another.

> **Note**: This guide requires {{site.kic_product_name}} version 3.2 or higher
> and the [_ReferenceGrant_](https://gateway-api.sigs.k8s.io/api-types/referencegrant/)
> resource definition from the [Gateway API](https://gateway-api.sigs.k8s.io/).
> _ReferenceGrant_ may not be available on all Kubernetes clusters.

## Example use case

Kim manages [consumers](/kubernetes-ingress-controller/{{page.release}}/reference/custom-resources/#kongconsumer)
for Example Organization. They have permission to the `qyzylorda` namespace and
creates consumers and credentials in it.

Luka manages Example Organization's `httpbin` Service in the `kualalumpur`

Check failure on line 26 in app/_src/kubernetes-ingress-controller/plugins/cross-namespace.md

View workflow job for this annotation

GitHub Actions / Vale

[vale] reported by reviewdog 🐶 [kong.Spelling] Did you really mean 'Luka'? Raw Output: {"message": "[kong.Spelling] Did you really mean 'Luka'?", "location": {"path": "app/_src/kubernetes-ingress-controller/plugins/cross-namespace.md", "range": {"start": {"line": 26, "column": 1}}}, "severity": "ERROR"}
namespace. Luka wants to apply different rate limits to different consumers,

Check failure on line 27 in app/_src/kubernetes-ingress-controller/plugins/cross-namespace.md

View workflow job for this annotation

GitHub Actions / Vale

[vale] reported by reviewdog 🐶 [kong.Spelling] Did you really mean 'Luka'? Raw Output: {"message": "[kong.Spelling] Did you really mean 'Luka'?", "location": {"path": "app/_src/kubernetes-ingress-controller/plugins/cross-namespace.md", "range": {"start": {"line": 27, "column": 12}}}, "severity": "ERROR"}
but they do not have full access to the `qyzylorda` namespace, and Kim does not
have access to the `kualalumpur` namespace. To manage these limits, Kim can
create _ReferenceGrants_ that permit Luka's _KongPlugins_ in `kualalumpur` to
interact with _KongConsumers_ in `qyzylorda` namespace.

## Create the test environment

Install [{{site.kic_product_name}} and the Gateway APIs](/kubernetes-ingress-controller/{{page.release}}/get-started/).

{:.note}
> This guide requires the Gateway API _ReferenceGrant_ resource, which
> is not available by default on all Kubernetes clusters.
> {{site.kic_product_name}} does support cross-namespace plugin configuration
> with the older Ingress API if ReferenceGrant is installed. This guide only
> demonstrates use of the Gateway API's HTTPRoute resource, but is generally
> applicable to Ingress as well. Example cross-namespace configuration in this
> guide interacts with Service and KongConsumer resources to be routing
> API-agnostic.

## Create namespaces

Create the two test namespaces:

```bash
kubectl create namespace qyzylorda
kubectl create namespace kualalumpur
```

The result should look like:

```text
namespace/qyzylorda created
namespace/kualalumpur created
```

## Create routes and services

Create a Service and Deployment in `kualalumpur`:

```bash
kubectl create -f https://docs.konghq.com/assets/kubernetes-ingress-controller/examples/httpbin-service.yaml -n kualalumpur
```

The result should look like:

```text
service/httpbin created
deployment.apps/httpbin created
```

Create an HTTPRoute to access the Service:

```bash
echo "
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: bintang
annotations:
konghq.com/strip-path: 'true'
spec:
parentRefs:
- name: kong
namespace: default
rules:
- matches:
- path:
type: PathPrefix
value: /bintang
backendRefs:
- name: httpbin
kind: Service
port: 80
" | kubectl apply -n kualalumpur -f -
```

The result should look like:
```text
httproute.gateway.networking.k8s.io/bintang created
```

Create a authentication plugin:

```bash
echo '
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: httpbin-basic-auth
config:
hide_credentials: true
plugin: basic-auth
' | kubectl apply -n kualalumpur -f -
```

The result should look like:
```text
kongplugin.configuration.konghq.com/httpbin-basic-auth created
```
and attach it to the Service:

```bash
kubectl annotate service httpbin -n kualalumpur konghq.com/plugins=httpbin-basic-auth
```

The result should look like:
```text
service/httpbin annotated
```

## Create consumers and credentials

1. Create Secrets to add `basic-auth` credentials for two consumers:

```bash
echo '
---
apiVersion: v1
kind: Secret
metadata:
name: aygerim-basic-auth
labels:
konghq.com/credential: basic-auth
stringData:
username: aygerim
password: aygerim-password
---
apiVersion: v1
kind: Secret
metadata:
name: rustem-basic-auth
labels:
konghq.com/credential: basic-auth
stringData:
username: rustem
password: rustem-password
' | kubectl apply -n qyzylorda -f -
```

The results should look like:
```text
secret/aygerim-basic-auth created
secret/rustem-basic-auth created
```

1. Create consumers named `aygerim` and `rustem` that use these credentials:

{% include /md/kic/consumer.md release=page.release name='aygerim' credName='aygerim-basic-auth' namespace='qyzylorda' %}

{% include /md/kic/consumer.md release=page.release name='rustem' credName='rustem-basic-auth' namespace='qyzylorda' %}

## Grant cross-namespace permissions

Luka and Kim wants to limit the amount of requests to the `httpbin` Service on

Check failure on line 181 in app/_src/kubernetes-ingress-controller/plugins/cross-namespace.md

View workflow job for this annotation

GitHub Actions / Vale

[vale] reported by reviewdog 🐶 [kong.Spelling] Did you really mean 'Luka'? Raw Output: {"message": "[kong.Spelling] Did you really mean 'Luka'?", "location": {"path": "app/_src/kubernetes-ingress-controller/plugins/cross-namespace.md", "range": {"start": {"line": 181, "column": 1}}}, "severity": "ERROR"}
a per-consumer basis. Because they only have access to their own namespaces,
they need to create rules that allow interactions across them.

By default, resources that support the [`konghq.com/plugins` annotation](/kubernetes-ingress-controller/{{page.release}}/reference/annotations/#konghqcomplugins)
can only attach plugins in their same namespace. ReferenceGrant resources can
grant permission to use plugins to resources in other namespaces.

To allow this cross-namespace interaction, Kim asks Luka to create a

Check failure on line 189 in app/_src/kubernetes-ingress-controller/plugins/cross-namespace.md

View workflow job for this annotation

GitHub Actions / Vale

[vale] reported by reviewdog 🐶 [kong.Spelling] Did you really mean 'Luka'? Raw Output: {"message": "[kong.Spelling] Did you really mean 'Luka'?", "location": {"path": "app/_src/kubernetes-ingress-controller/plugins/cross-namespace.md", "range": {"start": {"line": 189, "column": 53}}}, "severity": "ERROR"}
ReferenceGrant in `kualalumpur` that allows KongConsumers in `qyzylorda` to use
KongPlugins in `kualalumpur`:

```bash
echo '
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
name: qyzylorda-rate-limit
namespace: kualalumpur
spec:
from:
- group: "configuration.konghq.com"
kind: KongConsumer
namespace: qyzylorda
to:
- group: "configuration.konghq.com"
kind: KongPlugin
' | kubectl apply -f -
```

The results should look like:
```text
referencegrant.gateway.networking.k8s.io/qyzylorda-rate-limit created
```

Such ReferenceGrants allow the `from` resource (KongConsumers in the `qyzylorda`
namespace) to interact with KongPlugins in the ReferenceGrant's namespace
(`kualalumpur`).

## Create the cross-namespace plugins

With the ReferenceGrant in place, Luka can create the `rate-limiting`

Check failure on line 222 in app/_src/kubernetes-ingress-controller/plugins/cross-namespace.md

View workflow job for this annotation

GitHub Actions / Vale

[vale] reported by reviewdog 🐶 [kong.Spelling] Did you really mean 'Luka'? Raw Output: {"message": "[kong.Spelling] Did you really mean 'Luka'?", "location": {"path": "app/_src/kubernetes-ingress-controller/plugins/cross-namespace.md", "range": {"start": {"line": 222, "column": 35}}}, "severity": "ERROR"}
KongPlugins:

```bash
echo '
---
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: rate-limit-aygerim
annotations:
kubernetes.io/ingress.class: kong
config:
minute: 10
policy: local
plugin: rate-limiting
---
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: rate-limit-rustem
annotations:
kubernetes.io/ingress.class: kong
config:
minute: 5
policy: local
plugin: rate-limiting
' | kubectl apply -f - -n kualalumpur
```

The result should look like:
```text
kongplugin.configuration.konghq.com/rate-limit-aygerim created
kongplugin.configuration.konghq.com/rate-limit-rustem created
```

## Attach the plugins to their resources

{:.note}
> In practice, you would likely not manually apply plugin configuration first
> to the KongConsumers and second to the Services while they are live, as doing
> so creates a window of time where the limits apply to _all_ requests from
> those consumers, not just requests to the relevant Service. This guide does
> so to demonstrate creation of the necessary configuration in separate steps.
>
> In a real-world scenario, you should consider either managing configuration
> out of band (via externally-managed configuration deployed to the cluster via
> a continuous deployment system), during an outage window, or before the
> services or consumers are put into live use.

With the plugins in place, each namespace administrator can attach them to the
resources in their respective namespaces.

Kim attaches the plugins to the KongConsumers in `qyzylorda`. They use the
`<namespace>:<kongplugin-resource-name>` format in the plugin name to indicate
that the requested KongPlugin resides in another namespace:

```bash
kubectl annotate kongconsumer aygerim -n qyzylorda konghq.com/plugins=kualalumpur:rate-limit-aygerim
kubectl annotate kongconsumer rustem -n qyzylorda konghq.com/plugins=kualalumpur:rate-limit-rustem
```

The result should look like:

```text
kongconsumer.configuration.konghq.com/aygerim annotated
kongconsumer.configuration.konghq.com/rustem annotated
```

Luka attaches the plugins to the `httpbin` Service in `kualalumpur`:

Check failure on line 291 in app/_src/kubernetes-ingress-controller/plugins/cross-namespace.md

View workflow job for this annotation

GitHub Actions / Vale

[vale] reported by reviewdog 🐶 [kong.Spelling] Did you really mean 'Luka'? Raw Output: {"message": "[kong.Spelling] Did you really mean 'Luka'?", "location": {"path": "app/_src/kubernetes-ingress-controller/plugins/cross-namespace.md", "range": {"start": {"line": 291, "column": 1}}}, "severity": "ERROR"}

```bash
kubectl annotate service httpbin --overwrite -n kualalumpur konghq.com/plugins=httpbin-basic-auth,rate-limit-aygerim,rate-limit-rustem
```

Note that this command overrides the entire plugin annotation, so it needs to
include the `httpbin-basic-auth` plugin from earlier in the list, not just the
new rate limiting plugins.

The result should look like:

```text
service/httpbin annotated
```

## Test the configuration

Send a request using the `aygerim` consumer:

```bash
curl -sv $PROXY_IP/bintang/status/200 -u aygerim:aygerim-password 2>&1 | grep "X-RateLimit-Remaining-Minute"
```
The results should show the remaining limit expected for `aygerim`. Since
`aygerim` can send 10 requests per minute, and has sent 1, the response should
indicate there are 9 requests remaining:

```text
< X-RateLimit-Remaining-Minute: 9
```

Send a request using the `rustem` consumer:

```bash
curl -sv $PROXY_IP/bintang/status/200 -u rustem:rustem-password 2>&1 | grep "X-RateLimit-Remaining-Minute"
```
The results should show the remaining limit expected for `rustem`. Since
`rustem` can send 5 requests per minute, and has sent 1, the response should
indicate there are 4 requests remaining:

```text
< X-RateLimit-Remaining-Minute: 4
```
Loading