Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dex secret not found #155

Open
donydonald1 opened this issue Apr 18, 2024 · 18 comments
Open

dex secret not found #155

donydonald1 opened this issue Apr 18, 2024 · 18 comments
Assignees
Labels
bug Something isn't working

Comments

@donydonald1
Copy link

donydonald1 commented Apr 18, 2024

hello @khuedoan
i am a big fan of this and I have been trying to get work like this a week now but I have a little issue trying to make mine work. hoping you could help.
external secret didn't create a secret for dex.
please help

)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  dex
    Optional:    false
  kube-api-access-wl57w:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason  Age                    From     Message
  ----     ------  ----                   ----     -------
  Warning  Failed  60m (x12 over 62m)     kubelet  Error: secret "dex-secrets" not found
  Normal   Pulled  2m31s (x278 over 62m)  kubelet  Container image "ghcr.io/dexidp/dex:v2.38.0" already present on machine ``` 
@donydonald1 donydonald1 added the bug Something isn't working label Apr 18, 2024
@khuedoan
Copy link
Owner

khuedoan commented Apr 18, 2024

Hi, dex-secrets is created by https://github.com/khuedoan/homelab/blob/master/platform/dex/templates/secret.yaml, could you please post the output of:

kubectl describe -n dex externalsecret dex-secrets

@donydonald1
Copy link
Author

+ kubectl describe -n dex externalsecret dex-secrets
Name:         dex-secrets
Namespace:    dex
Labels:       argocd.argoproj.io/instance=dex
Annotations:  <none>
API Version:  external-secrets.io/v1beta1
Kind:         ExternalSecret
Metadata:
  Creation Timestamp:  2024-04-18T09:48:36Z
  Generation:          1
  Resource Version:    51172
  UID:                 94eb9cd1-310b-4a3d-8574-7ed4b326de5c
Spec:
  Data:
    Remote Ref:
      Conversion Strategy:  Default
      Decoding Strategy:    None
      Key:                  kanidm.dex
      Metadata Policy:      None
      Property:             client_id
    Secret Key:             KANIDM_CLIENT_ID
    Remote Ref:
      Conversion Strategy:  Default
      Decoding Strategy:    None
      Key:                  kanidm.dex
      Metadata Policy:      None
      Property:             client_secret
    Secret Key:             KANIDM_CLIENT_SECRET
    Remote Ref:
      Conversion Strategy:  Default
      Decoding Strategy:    None
      Key:                  dex.grafana
      Metadata Policy:      None
      Property:             client_secret
    Secret Key:             GRAFANA_SSO_CLIENT_SECRET
    Remote Ref:
      Conversion Strategy:  Default
      Decoding Strategy:    None
      Key:                  dex.gitea
      Metadata Policy:      None
      Property:             client_secret
    Secret Key:             GITEA_CLIENT_SECRET
  Refresh Interval:         1h
  Secret Store Ref:
    Kind:  ClusterSecretStore
    Name:  global-secrets
  Target:
    Creation Policy:  Owner
    Deletion Policy:  Retain
    Name:             dex-secrets
Status:
  Conditions:
    Last Transition Time:  2024-04-18T09:48:36Z
    Message:               could not get secret data from provider
    Reason:                SecretSyncedError
    Status:                False
    Type:                  Ready
Events:
  Type     Reason        Age                    From              Message
  ----     ------        ----                   ----              -------
  Warning  UpdateFailed  4m40s (x24 over 109m)  external-secrets  error retrieving secret at .data[0], key: kanidm.dex, err: secrets "kanidm.dex" not found 

this is also affecting other deployment as well and for some reasons none of the secrets generated works when trying to login to the deployments

woodpecker          pre-install-agent-secret-check-jsqrs                     0/1     Completed                    0                75m
woodpecker          woodpecker-agent-5b6945cc7b-8c49l                        0/1     CrashLoopBackOff             19 (2m41s ago)   75m
woodpecker          woodpecker-agent-5b6945cc7b-nrmmf                        0/1     CrashLoopBackOff             19 (2m52s ago)   75m

@kikokikok
Copy link

Same problem for me, I think the
kanidm.dex key is never creqted in the global-secrets ClusterSecretStore
Screenshot 2024-05-05 at 14 17 06

@khuedoan
Copy link
Owner

khuedoan commented May 7, 2024

kandim.dex should be created by default in the post install script, could you try running make post-install manually?

@kikokikok
Copy link

kikokikok commented May 7, 2024

Well the postscript fails when calling the reset of users with the python k8s client. It doesn't return the expected json payload on the stdout as expected which causes an error on json deserialization.
When executing with a remote ssh into the container, I see the json paylod

bash-5.2# make postinstall
make: *** No rule to make target 'postinstall'.  Stop.
bash-5.2# make post-install
Traceback (most recent call last):
  File "/home/cklat/homelab/./scripts/hacks", line 256, in <module>
    main()
  File "/home/cklat/homelab/./scripts/hacks", line 247, in main
    kanidm_login(["admin", "idm_admin"])
  File "/home/cklat/homelab/./scripts/hacks", line 158, in kanidm_login
    password = reset_kanidm_account_password(account)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cklat/homelab/./scripts/hacks", line 152, in reset_kanidm_account_password
    return json.loads(resp)['password']
           ^^^^^^^^^^^^^^^^
  File "/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 2 (char 1)

Manual bash inside the container:

kanidmd recover-account --output json admin
00000000-0000-0000-0000-000000000000 WARN     🚧 [warn]: This is running as uid == 0 (root) which may be a security risk.
00000000-0000-0000-0000-000000000000 WARN     🚧 [warn]: permissions on /data/server.toml may not be secure. Should be readonly to running uid. This could be a security risk ...
00000000-0000-0000-0000-000000000000 WARN     🚧 [warn]: WARNING: /data/server.toml has 'everyone' permission bits in the mode. This could be a security risk ...
00000000-0000-0000-0000-000000000000 WARN     🚧 [warn]: WARNING: /data/server.toml owned by the current uid, which may allow file permission changes. This could be a security risk ...
00000000-0000-0000-0000-000000000000 WARN     🚧 [warn]: WARNING: DB folder /data has 'everyone' permission bits in the mode. This could be a security risk ...
00000000-0000-0000-0000-000000000000 INFO     i [info]: Running account recovery ...
{"password":"VU29tSLcAqjccXWez12dQKhKNuPNWcJDcQ34NXK1gGGFSGwN"}

@numpythuckles
Copy link

numpythuckles commented Jun 28, 2024

Was running into the same JSON decode error on the reset_kanidm_account_password function.

Ran ./scripts/hacks outside of nix was able to create global-secrets for:

gitea.renovate
gitea.woodpecker

Renovate was able to create its own renovate-secret afterwards and began submitting PRs to Git.

An Oauth2 application was created in Gitea for Woodpecker also.

Still not seeing any kanidm.* in global-secrets. Working on it.

@eenikad
Copy link

eenikad commented Jul 16, 2024

@numpythuckles are you still working on this issue? Ran into the same problem and would appreciate every possible hint.

@brimdor
Copy link

brimdor commented Aug 5, 2024

getting this issue as well, any progress on it?

@brimdor
Copy link

brimdor commented Aug 21, 2024

Temporarily created this yaml that I kubectl apply when post-install fails in order to manually create the kanidm.dex
I place this in the scripts folder and called it kanidm_fix.yaml then ran kubectl apply -f scripts/kanidm_fix.yaml assuming this is being run from the root homelab folder.

The issue still persists though. The post-install errors every time. This temp fix only creates the secret but it does not apply the attributes in the other commands for the function that is failing.

apiVersion: batch/v1
kind: Job
metadata:
  name: secret-generator-kanidm-fix
  namespace: global-secrets
spec:
  backoffLimit: 3
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: secret-generator
          image: lachlanevenson/k8s-kubectl:latest
          command:
            - sh
            - -c
          args:
            - |
              apk add --no-cache openssl
              CLIENT_ID=$(openssl rand -base64 32)
              CLIENT_SECRET=$(openssl rand -base64 64)
              kubectl create secret generic kanidm.dex \
                --from-literal=client_id="${CLIENT_ID}" \
                --from-literal=client_secret="${CLIENT_SECRET}" \
                -n global-secrets
      serviceAccount: secret-generator

@lfleal
Copy link

lfleal commented Aug 22, 2024

Attempting to apply the above results in the following error:
error: resource mapping not found for name: "secret-generator-kanidm-fix" namespace: "global-secrets" from "scripts/kanidm_fix.yaml": no matches for kind "Job" in version "" ensure CRDs are installed first

@brimdor
Copy link

brimdor commented Aug 22, 2024

That's strange. I built and ran this after post-install failed. Did you run it at that point in the process?

@lfleal
Copy link

lfleal commented Aug 22, 2024

First time I ran it:
error: error validating "scripts/kanidm_fix.yaml": error validating data: apiVersion not set; if you choose to ignore these errors, turn validation off with --validate=false
Running with --validate=false generated the error above. The file is missing apiVersion, does that exist in your file?

@brimdor
Copy link

brimdor commented Aug 22, 2024

Ah ok, it was there but it was next to the code block indicators so it cut it off. Its fixed.

@lfleal
Copy link

lfleal commented Aug 22, 2024

Adding apiVersion: batch/v1 to the beginning of the yaml above fixes the error.

@brimdor
Copy link

brimdor commented Aug 22, 2024

Yep, as I stated, it is fixed.

@lfleal
Copy link

lfleal commented Aug 27, 2024

After getting the secrets added, I'm getting this now:
error parse config file /tmp/dex.config.yaml-461377176: error unmarshaling JSON: parse connector config: invalid character '\n' in string literal

@khuedoan
Copy link
Owner

khuedoan commented Sep 2, 2024

Thanks for reporting the issues everyone, I think the ./scripts/hacks have grown big and messy, so I'll rewrite it to make it a bit more reliable

@khuedoan
Copy link
Owner

khuedoan commented Sep 5, 2024

The following commits should fix the issue in the short term:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

7 participants