Issue with argocd integration

[AnthonyWC]

I am using argo-cd integration according to doc with viaductoss/ksops:v4.2.1 but i have issue with decryption; if i re-create kustomization.yaml files on argocd-repo-server and manually run kustomize it works so i am baffled why it doesn't work with argocd.

argocd@argocd-repo-server-57f4469b7b-s7zs4:~$ argocd version
argocd: v2.6.7+5bcd846
  BuildDate: 2023-03-23T14:57:27Z
  GitCommit: 5bcd846fa16e4b19d8f477de7da50ec0aef320e5
  GitTreeState: clean
  GoVersion: go1.18.10
  Compiler: gc
  Platform: linux/amd64

I am using GPG key from k8 secret; i imported gpg key in container command via

                gpg --import /sops-gpg/sops.asc &&
                gpg --list-key &&
                entrypoint.sh argocd-repo-server --redis argocd-redis:6379 --loglevel debug

where i mounted pgp secret and i see imported key listed.

:

argocd@argocd-repo-server-57f4469b7b-s7zs4:~$ gpg --list-keys
/home/argocd/.gnupg/pubring.kbx
-------------------------------
pub   rsa4096 2023-06-23 [SCEA]
      26BCC50767349385B6656762B0804DF66634080E

argocd@argocd-repo-server-57f4469b7b-s7zs4:~$ ls -la /usr/local/bin/k*
-rwxr-xr-x 1 root root 30840104 May 26 15:35 /usr/local/bin/ksops
-rwxr-xr-x 1 root root 27401720 May 26 15:36 /usr/local/bin/kustomize

cat << EOF > secret.yaml
apiVersion: v1
data:
  token: NEJEMUNFMjMtRTNDMC00QkJGLUE3NUUtQUJBMTAzRUU5NUM5
kind: Secret
metadata:
  name: app-secret
EOF

cat << EOF > .sops.yaml
creation_rules:
  - pgp: "26BCC50767349385B6656762B0804DF66634080E"
EOF

cat << EOF > kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

generators:
  - secret-generator.yaml
EOF

cat << EOF > secret-generator.yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: secret-generator
  annotations:
    config.kubernetes.io/function: |
        exec:
          path: ksops
files:
  - secret.yaml
EOF

sops --encrypt --in-place secret.yaml

file is encrypted with gpg:

argocd@argocd-repo-server-57f4469b7b-s7zs4:~/test$ head secret.yaml
apiVersion: ENC[AES256_GCM,data:mAM=,iv:mRn1QIHELRdKEGHMmuFGtkEMDEU+qAdxJlxiA2qTOyE=,tag:Fn3jC9x1OH4kx05KneITUA==,type:str]
data:
    token: ENC[AES256_GCM,data:4plLHBnsIrMVxntUA47kp7GrGfpJpofk4VlRDT3SWdWA53KudhSpGgOxLHBWMQT8,iv:HChdklSxbk5kFWGimu2dFXfNvii2Z6cIZ8eaW9dmQnA=,tag:vrqfVqyMcIDZ5qwNj71OVA==,type:str]
kind: ENC[AES256_GCM,data:uDVdhA32,iv:/baMnnz/x0wIX3Wx7FyaGkVqEYjm8356nRZ8RmLcd/Y=,tag:bg9owMMaiq7kYcIduTH7hA==,type:str]
metadata:
    name: ENC[AES256_GCM,data:lHa9oRGOD+AA1A==,iv:At7wPr8IUFzW2X4o0M58pwprjzqCqCHub5BbrsoH2DY=,tag:YYtwaUOVAvkyeRcXyumkYA==,type:str]

kustomize works as expected

argocd@argocd-repo-server-57f4469b7b-s7zs4:~/test$ kustomize build --enable-alpha-plugins --enable-exec .
apiVersion: v1
data:
  token: NEJEMUNFMjMtRTNDMC00QkJGLUE3NUUtQUJBMTAzRUU5NUM5
kind: Secret
metadata:
  name: app-secret

ksop-test2 directory contains same file and it works locally with kustomize build but fails in argocd:

argocd-repo-server-99bf848c4-wf2qx` argocd-repo-server time="2023-06-24T18:46:15Z" level=error 
msg="finished unary call with code Unknown" error="Manifest generation error (cached): `kustomize build 
.overlays/saas/testing3/deploy/ksop-test2 --enable-alpha-plugins --enable-exec` failed exit status 1: failed to 
evaluate function: error decrypting file \"secret.enc.yaml\" from manifest.Files: trouble decrypting file: Error getting 
data key: 0 successful groups required, got 0unable to generate manifests: error decrypting file \"secret.enc.yaml\" 
from manifest.Files: trouble decrypting file: Error getting data key: 0 successful groups required, got 0Error: 
couldn't execute function: exit status 1" grpc.code=Unknown grpc.method=GenerateManifest 
grpc.service=repository.RepoServerService grpc.start_time="2023-06-24T18:46:15Z" grpc.time_ms=125.063 
span.kind=server system=grpc

I had this as applicationset with git generator but i also tried manually creating it as app in GUI; both has issue decrypting file.

[devstein]

@AnthonyWC I don't know if this is still relevant, but my gut says this the argocd user in the pod doesn't have access to the mounted key

@dosu-bot what do you think?

[dosubot]

🤖

Hello @AnthonyWC,

Based on the information you've provided and the similar issues I found in the KSOPS repository, it seems like the problem might be related to the argocd user not having access to the mounted key.

Here are some steps you can take to troubleshoot this issue:

1. Ensure that the PGP key is available to the argocd user. This can be achieved by either copying the PGP key into the custom repo-server, as the argocd user import the PGP key, then delete the copied key or running a postStart script as the argocd user on the repo-server deployment that imports the key from a mount K8s secret.

2. Add the following environment variable to the container specification in the repo-server patch:

env:
    - name: GNUPGHOME
      value: /home/argocd/.gnupg

A working demo of the solution can be found at: https://github.com/james-callahan/example-gitops

Another potential issue could be the version mismatch of KSOPS. You mentioned that you are using KSOPS v4.2.1, but the latest version is v4.2.4. If you are using KSOPS v4.2.1, then the Dockerfile should be updated to reflect this.

Here is an example of how to update the Dockerfile:

# https://github.com/argoproj/argo-cd/blob/master/Dockerfile
ARG KSOPS_VERSION="v4.2.1"

#--------------------------------------------#
#--------Build KSOPS and Kustomize-----------#
#--------------------------------------------#

FROM viaductoss/ksops:$KSOPS_VERSION as ksops-builder

#--------------------------------------------#
#--------Build Custom Argo Image-------------#
#--------------------------------------------#

FROM argoproj/argocd:$ARGO_CD_VERSION

# Switch to root for the ability to perform install
USER root

ARG PKG_NAME=ksops

# Override the default kustomize executable with the Go built version
COPY --from=ksops-builder /usr/local/bin/kustomize /usr/local/bin/kustomize

# Add ksops executable to path
COPY --from=ksops-builder /usr/local/bin/ksops /usr/local/bin/ksops

# Switch back to non-root user
USER argocd

And the Argo CD Helm Chart should be updated as follows:

# Enable Kustomize Alpha Plugins via Argo CD ConfigMap, required for ksops
server:
  config:
    kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"

repoServer:
  # Use init containers to configure custom tooling
  # https://argoproj.github.io/argo-cd/operator-manual/custom_tools/
  volumes:
    - name: custom-tools
      emptyDir: {}

  initContainers:
    - name: install-ksops
      image: viaductoss/ksops:v4.2.1
      command: ["/bin/sh", "-c"]
      args:
        - echo "Installing KSOPS...";
          mv ksops /custom-tools/;
          mv kustomize /custom-tools/;
          echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
  volumeMounts:
    - mountPath: /usr/local/bin/kustomize
      name: custom-tools
      subPath: kustomize
    - mountPath: /usr/local/bin/ksops
      name: custom-tools
      subPath: ksops

This should ensure that the correct version of KSOPS is being used in your Argo CD integration.

Please try these suggestions and let me know if the issue persists.

Sources

• ArgoCD and GCP KMS not decrypting
• Trouble decrypting file
• Using PGP w/ ArgoCD
• README.md

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.