ksops with age using argocd

[nureyni]

Issue Summary

I am encountering an error when trying to decrypt a file using KSOPS with AGE in ArgoCD. The specific error occurs in the argocd-repo-server pod and relates to the decryption of the ./secrets/sops-age-privatekey-secret.yaml file. What i'm i doing wrong ?

Environment Details

• ArgoCD Version: v2.9.0
• KSOPS Version: v4.2.5

Context

1. Patched the argocd-repo-server deployment to install KSOPS and place the AGE private key in the container. The deployment YAML is as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
  namespace: argocd
spec:
  template:
    spec:
      # 1. Define an emptyDir volume which will hold the custom binaries
      volumes:
        - name: custom-tools
          emptyDir: {}
        - name: sops-age
          secret:
            secretName: sops-age
      # 2. Use an init container to download/copy custom binaries into the emptyDir
      initContainers:
        - name: install-ksops
          image: viaductoss/ksops:v4.2.5
          command: ["/bin/sh", "-c"]
          args:
            - echo "Installing KSOPS...";
              cp /usr/local/bin/ksops /custom-tools/;
              cp /usr/local/bin/kustomize /custom-tools/;
              echo "Done.";
          volumeMounts:
            - mountPath: /custom-tools
              name: custom-tools
      containers:
        - name: argocd-repo-server
          env:
            - name: SOPS_AGE_KEY_FILE
              value: /.config/sops/age/keys.txt
          volumeMounts:
            - mountPath: /usr/local/bin/kustomize
              name: custom-tools
              subPath: kustomize
            - mountPath: /usr/local/bin/ksops
              name: custom-tools
              subPath: ksops
            - mountPath: /.config/sops/age/keys.txt
              name: sops-age
              subPath: keys.txt
            - mountPath: /custom-tools
              name: custom-tools

2. Created a kustomization manifest to patch resources and added a secret generator using ./secret-generator.yaml with the following content:

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: secret-generator
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - ./secrets/age-privatekey-secret.sops.yaml
 

Expected Behavior

The expected behavior is for the KSOPS to successfully decrypt the file specified in ./secrets/age-privatekey-secret.sops.yaml and add it to the cliuster at the namespace argocd. Not that ksops is actually installed and the private key in place in the container (have done test using the container shell)

Actual Behavior

The argocd-repo-server pod is throwing the following error:

time="2023-11-12T16:55:16Z" level=error msg="`kustomize build /tmp/_argocd-repo/fcb3ee92-7743-44c2-9169-a7ac6f4b42a1/bootstrap/argo-cd --enable-alpha-plugins --enable-exec` failed exit status 1: failed to evaluate function: error decrypting file \"./secrets/sops-age-privatekey-secret.yaml\" from manifest.Files: trouble decrypting file: Error getting data key: 0 successful groups required, got 0unable to generate manifests: error decrypting file \"./secrets/sops-age-privatekey-secret.yaml\" from manifest.Files: trouble decrypting file: Error getting data key: 0 successful groups required, got 0Error: couldn't execute function: exit status 1" execID=1da4c
time="2023-11-12T16:55:16Z" level=info msg=Trace args="[kustomize build /tmp/_argocd-repo/fcb3ee92-7743-44c2-9169-a7ac6f4b42a1/bootstrap/argo-cd --enable-alpha-plugins --enable-exec]" dir= operation_name="exec kustomize" time_ms=1381.548711

Additional Information

creation_rules:
  - path_regex: .*\.sops\.ya?ml
    age: age1dsm5k7vngkdghdtkygfttcwtjfxdh9ks7tmz7m55h22wmr0vwd2s526wj7
    encrypted_regex: "^(data|stringData)$"

and configmaps

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
  labels:
    app.kubernetes.io/name: argocd-cmd-params-cm
    app.kubernetes.io/part-of: argocd
data:
  reposerver.log.level: "debug"

---

Attachments:

My encrypted file and the project files in the screenshot below

[dosubot]

🤖

Hello @nureyni! I'm Dosu, an AI here to assist with your issue while we wait for a human maintainer. I'm here to help troubleshoot bugs, answer questions, and guide you through contributing to this project. I'm currently reviewing the issue you've posted and will provide a more detailed response shortly. Hang tight!

[dosubot]

🤖

From the error message you provided, it seems like the decryption process is failing. This could be due to several reasons:

1. File not found: The file you are trying to decrypt might not exist in the specified path. This could be due to a typo in the file name or the file not being present in the expected location. You can check this in the decryptFile function where the file is read using ioutil.ReadFile(file).

2. Incorrect file format: The file format might not be supported by the SOPS for decryption. This can be checked in the decryptFile function where the file format is determined using formats.FormatForPath(file).

3. Decryption failure: The decryption process itself might be failing. This could be due to an incorrect or missing decryption key, or the file might not have been encrypted correctly in the first place. This can be checked in the decryptFile function where the decryption is performed using decrypt.DataWithFormat(b, format).

4. Incorrect file structure: The structure of the file might not be as expected, causing the unmarshalling to fail. This can be checked in the generate function where the file content is unmarshalled using yaml.Unmarshal(raw, &manifest).

5. Missing 'files' or 'secretFrom' key: The 'files' or 'secretFrom' key might be missing in the ksops manifests. This can be checked in the generate function where the presence of these keys is checked.

6. In the context shared, the file seems to be encrypted using PGP instead of AGE. If you are trying to decrypt it using an AGE key, it will fail. You need to ensure that the file is encrypted using the same method (AGE or PGP) as the decryption method you are using.

To resolve the issue, you should check the logs of the argocd-repo-server pod for any error messages that might give a clue as to what is going wrong. You should also verify that the file exists, is in the correct format, and is correctly encrypted with the correct key. If the issue persists, you might need to debug the KSOPS code to find out exactly where the error is occurring.

You can refer to the KSOPS code here for more details.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Sources

• kustomize build .age --enable-alpha-plugins` failed exit status 1 trouble decrypting file Error getting data key
• Trouble decrypting file
• ksops.go
• test/legacy/binaryfile/secret.enc.yaml
• test/krm/binaryfile/secret.enc.yaml
• test/krm/metadata/secret.enc.env

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[devstein]

Hi @nureyni thanks for making an issue. Are you able to build the manifests when you exec into the pod?

The error indicates KSOPS can't find the age key. Typically this occurs when the user argocd doesn't have permission to read the mounted key. Have you confirmed argocd can read the mounted age file and directory?

[nureyni]

Hi @devstein,

Thanks for your reply

Below the content of my deployment (i removed all unrelevent keys):

apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-repo-server
  template:
    metadata:
      labels:
        app.kubernetes.io/name: argocd-repo-server
    spec:
      volumes:
        - name: custom-tools
          emptyDir: {}
        - name: sops-age
          secret:
            secretName: sops-age
            defaultMode: 420
        - name: ssh-known-hosts
          configMap:
            name: argocd-ssh-known-hosts-cm
            defaultMode: 420
        - name: tls-certs
          configMap:
            name: argocd-tls-certs-cm
            defaultMode: 420
        - name: gpg-keys
          configMap:
            name: argocd-gpg-keys-cm
            defaultMode: 420
        - name: gpg-keyring
          emptyDir: {}
        - name: tmp
          emptyDir: {}
        - name: helm-working-dir
          emptyDir: {}
        - name: var-files
          emptyDir: {}
        - name: plugins
          emptyDir: {}
      initContainers:
        - name: install-ksops
          image: viaductoss/ksops:v4.2.5
          command:
            - /bin/sh
            - '-c'
          args:
            - >-
              echo "Installing KSOPS..."; cp /usr/local/bin/ksops
              /custom-tools/; cp /usr/local/bin/kustomize /custom-tools/; echo
              "Done.";
          resources: {}
          volumeMounts:
            - name: custom-tools
              mountPath: /custom-tools
      containers:
        - name: argocd-repo-server
          image: quay.io/argoproj/argocd:v2.9.0
          args:
            - /usr/local/bin/argocd-repo-server
          ports:
            - containerPort: 8081
              protocol: TCP
            - containerPort: 8084
              protocol: TCP
          env:
            - name: SOPS_AGE_KEY_FILE
              value: /.config/sops/age/keys.txt
          resources: {}
          volumeMounts:
            - name: custom-tools
              mountPath: /usr/local/bin/kustomize
              subPath: kustomize
            - name: custom-tools
              mountPath: /usr/local/bin/ksops
              subPath: ksops
            - name: sops-age
              mountPath: /.config/sops/age/keys.txt
              subPath: keys.txt
            - name: custom-tools
              mountPath: /custom-tools
            - name: ssh-known-hosts
              mountPath: /app/config/ssh
            - name: tls-certs
              mountPath: /app/config/tls
            - name: gpg-keys
              mountPath: /app/config/gpg/source
            - name: gpg-keyring
              mountPath: /app/config/gpg/keys
            - name: argocd-repo-server-tls
              mountPath: /app/config/reposerver/tls
            - name: tmp
              mountPath: /tmp
            - name: helm-working-dir
              mountPath: /helm-working-dir
            - name: plugins
              mountPath: /home/argocd/cmp-server/plugins
          securityContext:
            capabilities:
              drop:
                - ALL
            runAsNonRoot: true
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            seccompProfile:
              type: RuntimeDefault
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: argocd-repo-server
      serviceAccount: argocd-repo-server
      automountServiceAccountToken: false
      securityContext: {}
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

inside the container argocd-repo-server, a cat $SOPS_AGE_KEY_FILE return the content of the key like this :

# created: 2023-11-09T22:23:46+01:00
# public key: age1dsm5k********************************
AGE-SECRET-KEY-1UJTK80P5HZLR8S5T****************

[nureyni]

@devstein do you have any idea on how can ksops find the AGE key

[blinkeye]

Hi @nureyni, it's unclear to me if you're trying to use an age key which is still an encrypted secret or if you actually have two age keys (one is your own, one is used for argocd to decrypt your resources).

The error:

trouble decrypting file: Error getting data key: 0 successful groups required, got 0
unable to generate manifests: error decrypting file \"./secrets/sops-age-privatekey-secret.yaml\" from manifest.

means there are no age keys available that can decrypt the file, not that $SOPS_AGE_KEY_FILE can't be read. You can reproduce that locally by setting export SOPS_AGE_KEY_FILE=/tmp/dummy.txt and trying to decrypt an encrypte file - you'll get the same error if there's no valid key in that file. Best use the age key you use on your target pod to validate.

IMHO you should have at least two age keys listed in your .sops.yaml (seperated by ,):

creation_rules:
  - path_regex: .*\.sops\.ya?ml
    age: "age1***this-is-your-personal-key,age1***this-is-argocd-key-to-decrypt-other-manifests"
    encrypted_regex: "^(data|stringData)$"

You'll want to encrypt the secrets/sops-age-privatekey-secret.yaml as you do today with your private key. You can then deploy the argocd manifests to your target namespace (the ./secrets/sops-age-privatekey-secret.yaml will be decrypted with your own age key). For any other resources argocd is managing you'll want to create secrets which are decryptable with both your own private age key and the one argocd will be using, so you'll end up with multiple recipients in the metadata of the encrypted file.

To confirm: in your screenshot we see that ./secrets/sops-age-privatekey-secret.yaml has a single recipient listed as age1helqc... which I asume is your private one - but this doesn't match the one from argocd, $SOPS_AGE_KEY_FILE lists the public key as age1dsm5k...

[Bear-LB]

Had the same error but it was due to me having environment variable set directly to the age key string instead of a file path that contains the age key

[tcassaert]

I had the same error. I had both the env var containing the age key and the env var containing a path to the keyfile. Apparently that doesn't work. Only the env var containing the path to the keyfile is necessary, adding both makes it fail.

[freeo]

More people will fall into this trap. The core issue is the incomplete documentation in the README.md for ArgoCD+Helm Chart.

After way too many tries I arrived at this config:

argocd-values.yaml

configs:
  cm:
    kustomize.buildOptions: "--enable-helm --enable-alpha-plugins --enable-exec"

repoServer:
  # Use init containers to configure custom tooling
  # https://argoproj.github.io/argo-cd/operator-manual/custom_tools/
  volumes:
    - name: custom-tools
      emptyDir: {}
    - name: sops-age
      secret:
        secretName: sops-age

  initContainers:
    - name: install-ksops
      image: viaductoss/ksops:v4.3.2
      command: ["/bin/sh", "-c"]
      args:
        - echo "Installing KSOPS...";
          mv ksops /custom-tools/;
          mv kustomize /custom-tools/;
          echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
  volumeMounts:
    - mountPath: /usr/local/bin/kustomize
      name: custom-tools
      subPath: kustomize
    - mountPath: /usr/local/bin/ksops
      name: custom-tools
      subPath: ksops
      # missing in official docs until EOF
    - mountPath: /home/argocd/.config/kustomize/plugin/viaduct.ai/v1/ksops/ksops
      name: custom-tools
      subPath: ksops
    - mountPath: /.config/sops/age/key.txt
      name: sops-age
      subPath: key.txt

  env:
    - name: SOPS_AGE_KEY_FILE
      value: /.config/sops/age/key.txt

Side note @tcassaert : When I experimented locally, I needed to set both env vars for it to work! But in the argocd-repo-server the path to the key file was enough, just as you wrote. Don't understand why ksops is behaving differently locally.

export AGEKEY=/path/to/my.agekey

# 1. Both: WORKS
export SOPS_AGE_KEY_FILE=$AGEKEY
export SOPS_AGE_KEY=$(cat $AGEKEY)

# 2. Content Only: doesn't work
export SOPS_AGE_KEY_FILE=
export SOPS_AGE_KEY=$(cat $AGEKEY)

# 3. Path only: doesn't work
export SOPS_AGE_KEY_FILE=$AGEKEY
export SOPS_AGE_KEY=