Watch AWS for version changes ?

[derrickburns]

Would it make sense to watch the SecretsManager for version changes?

[mumoshu]

Hey! What's your goal, and expected use-cases?

Is that to reconcile respective K8s secrets accordingly? If so, how would your app that mounting the K8s secret as envvars or a container volume reloads it?

[maxbrunet]

Hey, I was thinking about the same thing too.

@mumoshu How do you suggest to reload them?

At the moment, I guess I would need 3 steps to update a deployment:

• Change secret in SecretsManager
• Update VersionId in AWSSecret
• Trigger a rolling restart of the deployment(s) (e.g. update the kubernetes.io/change-cause annotation) (I'm not even sure how to identify them accurately yet)

Reconcile K8s secrets with SecretsManager would simplify the process by removing a step.

I only use AWSSecrets with Cronjobs for now (no need to relaod). I use Kustomize's secretGenerator for deployments, since it suffixes the secret's name with its hash, which changes and triggers a rolling update.

Thanks for your work!

[mumoshu]

@maxbrunet Hey!

Reconcile K8s secrets with SecretsManager would simplify the process by removing a step.

I guess no. The suggested proposal seems to update the K8s secret without recreating it. And your k8s deployment has no way to react on changes in the K8s secret.

Perhaps you want a CI or CD pipeline that rolling-updates your K8s deployment each time the upstream SecretsManager secret changes?

[maxbrunet]

The suggested proposal seems to update the K8s secret without recreating it. And your k8s deployment has no way to react on changes in the K8s secret.

Exactly.

Perhaps you want a CI or CD pipeline that rolling-updates your K8s deployment each time the upstream SecretsManager secret changes?

Yes, it's the only option I've thought of so far.

Reconcile the K8s secret with the SecretsManager would be pretty smooth for Cronjobs. And for Deployments, I know that I need to rolling-updates them when I make a change in the SecretsManager, having to update the VersionId in the AWSSecret object adds an uncessarry extra step. Don't you agree?

[mumoshu]

I mostly agree, but there's a pitfall.

When this feature is used with Deployment or CronJob, you don't really have a control of when the changes to secrets are "applied" to pods. One such example is that when one of your pods managed by either Deployment or CronJob crashed after a secret has changed, the new pod automatically created as replacement receives the secret value after the change.

IMHO the only perfect solution I've found so far is create a K8s secret per a SSM secret version. You keep at least two recent versions of your K8s secrets. Trigger a rolling-update by changing the K8s secret name referenced from your deployment's pod spec. This way you have a complete control of when to apply changes, plus the ability to rollback. I.e. do things declaratively.

That said, it isn't "unnecessary". Making your workflow complicated isn't nice, though. So probably we need an another way to make the workflow simpler?

What if aws-secret-operator added an ability to automatically create a new K8s secret per a new AWS secret? Then all you have to do is update Deployment or CronJob to refer the new K8s secret created for the new secret version.

---

You'd want to automate it further by automatically updating Deployment or CronJob per a new K8s secret. But that's an another story.

Implementing the automation in aws-secret-operator operator seems feasible. But then there's an another problem - you have two things that updates your Deployment or CronJob, your CI/CD pipeline and aws-secret-operator. That said, perhaps you'll like your CI/CD pipeline combined with aws-secret-operator with the feature to automatically create a new k8s secret per a AWS secret version?

WDYT?

[ecout]

@mumoshu
1.-Isn't this issue a duplicate?
#25

See link for details as to why it works, in case someone doesn't know GO SDK will always return the LATEST version of the secret if neither Stage nor Version ID are specified. Luckily you haven't really ENFORCED specifying version ID so that's true and tested ATM.

2.-I've been doing quite some reading on this and k8s Secrets in volumes as of 1.14 (AFAIK) AUTOMATICALLY update values in the POD if the Secret itself is updated.

Go to the bottom of this section:
https://v1-14.docs.kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod

@maxbrunet Please see the link at the top, there's no need for messing with the version ID GUID.

Edit:
3.- From a POD perspective Devs need to implement a watcher/poller to check for changes in Volume files/ENV variables in the POD and update the Container's app memory with the latest values.