Kubernetes OIDC auth not working

[jsleeio]

We have setup Kubernetes OIDC authentication against our Google domain. Note, this is not a GKE cluster like in #65; we're hosted in AWS. Our OIDC setup uses k8s-oidc-helper.

I don't understand the auth bits sufficiently to debug this. For now I can work around it by retaining a non-OIDC context in my $HOME.kube/config and using that for out-of-cluster development with pykube, then deploying my apps in the cluster with a serviceaccount.

OIDC user in kubeconfig:

(sorry about all the redacting... I can only assure you that the setup works with kubectl!)

users:
- name: REDACTED
  user:
    auth-provider:
      config:
        client-id: REDACTED.apps.googleusercontent.com
        client-secret: REDACTED
        extra-scopes: groups
        id-token: REDACTED
        idp-issuer-url: https://accounts.google.com
        refresh-token: REDACTED
      name: oidc

kubectl "just works" in all of our contexts:

$ kubectl get ing -n core -o jsonpath='{..name}'
grafana-grafana logging-graylog logging-kopf

test code

$ cat kubetest.py
#!/usr/bin/env python3

import os
import pykube

kubeconfig = os.path.join(os.environ['HOME'], '.kube', 'config')
config = pykube.KubeConfig.from_file(kubeconfig)
api = pykube.HTTPClient(config)

watch = pykube.Ingress.objects(api, namespace='core')
for watch_event in watch.watch():
    print(watch_event)

works OK with a context that is not using OIDC:

$ kubectl config use-context non.oidc.context
Switched to context "non.oidc.context".
$ ./kubetest.py
WatchEvent(type='ADDED', object=<Ingress grafana-grafana>)
WatchEvent(type='ADDED', object=<Ingress graylog-kopf>)
WatchEvent(type='ADDED', object=<Ingress graylog-graylog>)

but with OIDC, fails:

$ kubectl config use-context oidc.context
Switched to context "oidc.context".
$ ./kubetest.py
Traceback (most recent call last):
  File "./kubetest.py", line 16, in <module>
    for watch_event in watch.watch():
  File "/usr/local/lib/python3.6/site-packages/pykube/query.py", line 158, in object_stream
    r = self.api.get(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 127, in get
    return self.session.get(*args, **self.get_kwargs(**kwargs))
  File "/usr/local/lib/python3.6/site-packages/pykube/http.py", line 39, in session
    self._session = build_session(self.config, self.gcloud_file)
  File "/usr/local/lib/python3.6/site-packages/pykube/session.py", line 23, in build_session
    s = _session_object("gcp", config, gcloud_file)
  File "/usr/local/lib/python3.6/site-packages/pykube/session.py", line 47, in _session_object
    return GCPSession(config, gcloud_file).create()
  File "/usr/local/lib/python3.6/site-packages/pykube/session.py", line 85, in __init__
    self.credentials = self._load_default_gcloud_credentials()
  File "/usr/local/lib/python3.6/site-packages/pykube/session.py", line 166, in _load_default_gcloud_credentials
    raise PyKubeError('Google cloud well known file missing, configure your gcloud session')
pykube.exceptions.PyKubeError: Google cloud well known file missing, configure your gcloud session

installed pykube from latest master commit in a virtualenv and retested with OIDC:

$ ./kubetest.py
Traceback (most recent call last):
  File "./kubetest.py", line 11, in <module>
    for watch_event in watch.watch():
  File "/Users/johnslee/vc/sm/kube2pingdom/venv/src/pykube/pykube/query.py", line 160, in object_stream
    self.api.raise_for_status(r)
  File "/Users/johnslee/vc/sm/kube2pingdom/venv/src/pykube/pykube/http.py", line 234, in raise_for_status
    resp.raise_for_status()
  File "/Users/johnslee/vc/sm/kube2pingdom/venv/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://oidc.context.api.endpoint/apis/extensions/v1beta1/namespaces/core/ingresses?watch=true

[nielsole]

You should be able to circumvent that by using kubectl proxy and then connecting to localhost