oauth2-proxy and programmatic access clients

[kromanow94]

Description

This is a follow up for #2409.

It was discussed that the oauth2-proxy integration with Istio should also cover m2m setup with authorization bearer tokens issued by oidc provider with emphasis on also supporting the default kubernetes oidc served behind https://kubernetes.default.svc.cluster.local.

The exact setup and reasoning was described it this and following comments: #2409 (comment).

oauth2-proxy should be configured to skip jwt validation when a specific issuer=audience pair is provided with extra_jwt_issuers configuration option. This pair will be https://kubernetes.default.svc.cluster.local=https://kubernetes.default.svc.cluster.local for the default k8s oidc. This will forward the traffic directly to Istio and then we can have a RequestAuthentication resource that will verify the JWT and authorize access to the services based on JWT Payload.

As mentioned in the #2409, I'll provide a PR for this feature.

Considerations

Istio

When using the default k8s oidc (which k8s configures by default if not provided with external oidc client details), the issuer is somewhat problematic. This is because the issuer is served behind self-signed certificates and accessible under endpoint that requires authentication.

Configuring traffic with RequestAuthentication for the default k8s oidc like below will not work and istiod will output following log:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: kubernetes-jwt
  namespace: istio-system
spec:
  jwtRules:
    - forwardOriginalToken: true
      issuer: https://kubernetes.default.svc.cluster.local

2023-08-29T14:46:13.084271Z     info    model   The JWKS key is not yet fetched for issuer https://kubernetes.default.svc.cluster.local/ (), using a fake JWKS for now

Following the log about JWKS and knowing the detail about cert and auth of oidc issuer, I was able to verify that if we were to take the JWKS and provide it directly as string in RequestAuthentication, this issue is not happening anymore.

Example

$ curl https://kubernetes.default.svc.cluster.local/.well-known/openid-configuration -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" | jq
{
  "issuer": "https://kubernetes.default.svc.cluster.local/",
  "jwks_uri": "https://100.65.51.192:6443/openid/v1/jwks",
  "response_types_supported": [
    "id_token"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ]
}

$ curl https://kubernetes.default.svc.cluster.local/openid/v1/jwks -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" -k
{"keys": ...}

apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
  name: kubernetes-jwt
  namespace: istio-system
spec:
  jwtRules:
  - forwardOriginalToken: true
    issuer: https://kubernetes.default.svc.cluster.local/
    jwks: '{"keys": ...}'
    outputClaimToHeaders:
    - claim: sub
      header: x-auth-request-user

$ curl httpbin.httpbin/headers -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
{
  "headers": {
    "Accept": "*/*", 
    "Authorization": "Bearer ...",
    "Host": "httpbin.httpbin", 
    "User-Agent": "curl/8.2.1", 
    "X-Auth-Request-User": "system:serviceaccount:default:default"
  }
}

oauth2-proxy

oauth2-proxy has to be configured with following config to skip verification of JWTs when issued by kubernetes oidc:

skip_jwt_bearer_tokens = true
extra_jwt_issuers = "https://kubernetes.default.svc.cluster.local=https://kubernetes.default.svc.cluster.local"

It's important to know that although the auth process will be skipped for a request with a jwt bearer token, oauth2-proxy will still verify the jwt against the oidc to verify if the token was indeed created by the issuer. Because of that it's still important to ensure that the self signed certs of default k8s oidc issuer are trusted in oauth2-proxy container.

I verified that if oauth2-proxy container has content of /run/secrets/kubernetes.io/serviceaccount/ca.crt in /etc/ssl/certs/ca-certificates.crt, the certs of https://kubernetes.default.svc.cluster.local will be trusted. We could provide an init container that prepares the certificates.crt in emptyDir volume and then it would be mounted in /etc/ssl/certs/.

Another thing is with configuration of anonymous access to the oidc endpoints. This could be simply achieved with:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: anonymous-oidc-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:service-account-issuer-discovery
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:unauthenticated

The ClusterRole system:service-account-issuer-discovery is created by default by kubernetes. It was verified in EKS, vcluster and dind that it exist.

$ kubectl get clusterroles system:service-account-issuer-discovery -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2023-08-30T10:47:38Z"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:service-account-issuer-discovery
  resourceVersion: "811"
  uid: 23c252dd-eab5-4896-80ca-56eced0bf4fb
rules:
- nonResourceURLs:
  - /.well-known/openid-configuration
  - /openid/v1/jwks
  verbs:
  - get

I'd appreciate some comment to confirm that the described method of achieving m2m tokens with oauth2-proxy is accepted.

@kimwnasptd @juliusvonkohout @axel7083

[kromanow94]

This should be addressed after #2516