Secret Store Add-on: Problem reading key/value kind of secret from AWS Secret Manager

[jagadeeshmaneri]

Describe the bug

We have a helm chart configuration that reads the secrets mounted on to K8s cluster. Using EKS blueprints we created the nodes/pods from those helm charts on an EKS cluster. The helm charts expect the secret name to be my-secret-name and look for different keys in different chart definitions across the project like my-secret-key-1, my-secret-key-2, etc.,

All these keys and values (secrets) are saved in AWS Secret Manager like

{
  "my-secret-key-1": "key-1-value",
  "my-secret-key-2": "key-2-value",
  ...
}

Note: The code snippet of EKS Blueprints and helm charts is in the Reproduction Steps section. Please check

The secret team, secret provider class, cluster, etc., gets created and the secret also gets mounted with this setup. The problem exists in the way/format of the secret that got mounted.

Expected Behavior

When the kube config is updated and kubectl get secret -n my-namespace command is run, the output is expected to have the

NAME                  TYPE                                  DATA   AGE
default-token-rx4zp   kubernetes.io/service-account-token   3      5d22h
my-secret-name        Opaque                                37     44h

37 is the number of keys we have in secret

And when you describe the secret all the keys should be listed like

Name:         my-secret-name
Namespace:    my-namespace
Labels:       cattle.io/creator=norman
Annotations:  field.cattle.io/creatorId: u-kqxyaw6tcm
              ...

Type:  Opaque

Data
====
my-secret-key-1:              26 bytes
my-secret-key-2:              12 bytes
...
...

And when the secret is read in helm charts, it should just return the value of that secret key.

Current Behavior

kubectl get secret -n my-namespace is returning

NAME                  TYPE                                  DATA   AGE
default-token-rx4zp   kubernetes.io/service-account-token   3      5d22h
my-secret-name        Opaque                                1      44h

The issue here is that all the key/value items are coming as one object (stored as JSON) as you see in the DATA column of first out above and the kuebctl describe secret/my-secret-name -n my-namespace is returning only 1 item.

Name:         my-secret-name
Namespace:    my-namespace
Labels:       cattle.io/creator=norman
Annotations:  field.cattle.io/creatorId: u-kqxyaw6tcm
              ...

Type:  Opaque

Data
====
my-secret-key-1:              2677 bytes

If we have multiple items in the data section of the CDK code (TeamApp class above), the describe secret command, the result will repeat those many items with the same size (2677 bytes). See below

Name:         my-secret-name
Namespace:    my-namespace
Labels:       cattle.io/creator=norman
Annotations:  field.cattle.io/creatorId: u-kqxyaw6tcm
              ...

Type:  Opaque

Data
====
my-secret-key-1:              2677 bytes
my-secret-key-2:              2677 bytes

Reproduction Steps

export class BluePrintsCluster extends Stack {
  constructor(scope: Construct, id: string, props: DeployStackProps) {
    super(scope, id, props);

    const clusterName = `${props.stackName}-learning`;
    const repoUrl = 'https://github.com/Organization/helmcharts-repo.git';
    const addOns: Array<blueprints.ClusterAddOn> = [
      new blueprints.addons.VpcCniAddOn(),
      new blueprints.addons.SecretsStoreAddOn(),
      new ArgoCDAddOn({
        bootstrapRepo: {
          repoUrl,
          credentialsSecretName: "github-token",
          credentialsType: "TOKEN",
          targetRevision: 'branch-name',
          path: "apps",
        },
        adminPasswordSecretName: "adminPasswordSecretName",
      }),
      new blueprints.addons.AwsLoadBalancerControllerAddOn(),
      new blueprints.addons.EfsCsiDriverAddOn(),
    ];

    blueprints.EksBlueprint.builder()
      .addOns(...addOns)
      .account(props.env?.account)
      .region(props.env?.region)
      .teams(new TeamApp(this))
      .build(this, clusterName);
  }
}

export class TeamApp extends ApplicationTeam {
  constructor(scope: Construct) {
    super({
      name: 'app',
      namespace: 'my-namespace',
      teamSecrets: [
        {
          secretProvider: new blueprints.LookupSecretsManagerSecretByName(
           'my-secret-name', 
          ),
          kubernetesSecret: {
            secretName: 'my-secret-name',
            type: blueprints.KubernetesSecretType.OPAQUE,
            data: [
              {
                key: 'my-secret-key-1'
              }
            ]
          },
        },
      ],
    });
  }
}

Part of the helm chart code

      # Secrets spec definition
 
      serviceAccountName: app-sa
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: app-aws-secrets
      
      # mounting secrets-store as a volume
      volumeMounts:
          - name: secrets-store-inline
            mountPath: /mnt/secrets-store
            readOnly: true

      // secret is read like
      customSecret:
        SECRET_1:
          my-secret-name: my-secret-key-1

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.37.1

EKS Blueprints Version

1.2.0

Node.js Version

16.6.0

Environment details (OS name and version, etc.)

Linux

Other information

No response

[shapirov103]

Hello @jagadeeshmaneri , my apologies for not replying sooner.

The reason why this was missed is because it does not appear to be a blueprints issue. Rather it is a feature of CSI Secret Store driver.

My understanding is that you have a multi-key secret in AWS and you are trying to produce a multi-key secret in EKS. The CSI Secret Store driver is very flexible, to the point when it is not quite intuitive. It was designed to be able to pull a single secret from AWS and map it to several secrets in Kubernetes or a single secret in different configurations.

Here is the shape of a SecretStoreProviderClass that creates a multi-key secret in EKS:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "arn:aws:secretsmanager:us-east-2:111122223333:secret:MySecret-a1b2c3"
        jmesPath: 
            - path: username
              objectAlias: dbusername
            - path: password
              objectAlias: dbpassword

Please refer to the official documentation for more info.

When defining team secrets with EKS Blueprints you can use the jmesPath array to provide mapping between keys in the AWS Secret and Kubernetes secret. Example below takes a secret with three keys (username, password, url) and maps them to a secret in k8s

return {
        secretProvider: new LookupSecretsManagerSecretByName(secretName),
        jmesPath: [{ path: "url", objectAlias: "url" }, { path: "username", objectAlias: "username" }, { path: "password", objectAlias: "password" }],
        kubernetesSecret: {
            secretName: secretName,
            labels: {"argocd.argoproj.io/secret-type": "repo-creds"},
            data: [
                { key: "url", objectName: "url" },
                { key: "username", objectName: "username" },
                { key: "password", objectName: "password" }
            ]
        }
    };

Note that the data attribute is also very important. The objectName in the data mapping corresponds to the objectAlias in jmesPath. Hope it helps, and I agree that this particular behavior may not be the most evident to the customers.

[elamaran11]

@jagadeeshmaneri Do you still have concerns with this issue?