add secrets config and vault secret actions

[kelkawi-a]

This PR introduces the following changes and additions:

• Prior to this, users would attach an env-file resource to specify a set of environment variables which should be injected into the workload container and used by the workflow definitions. This approach was problematic in that it handles secret credentials poorly. The env-file is now deprecated in favor of the secrets config described below.
• Adds the secrets config parameter, which can be used to instruct the charm to fetch secrets from three sources and inject them as environment variables into the workload container:
  • env: These are plaintext environment variables, which are usually not secret.
  • juju (Requires Juju 3.3+): These are variables stored as Juju secrets, where the user provides the ID of the secret and the key from which to fetch the value.
  • vault: These are variables fetched from Vault, provided the charm is related to a Vault charm.
• Removes the logic where the charm was injecting Vault connection parameters into the workload container, as the workflows ideally do not need to interact with Vault to fetch secrets. This work is now handled by the charm, the workload container should only expect specific environment variables to be present.
• Adds actions for adding and reading secrets from Vault.
• Updates docs and tests to reflect the changes.

Thanks to @gtato for the suggestions and useful design discussions.

[AmberCharitos]

Great work! This is really nice, some non-blocking suggestions and one important question around the use of temporal actions to add and remove vault secrets.

[AmberCharitos]

Question: I'm a bit confused as to why we would add secrets to vault via Temporal - is there no way to do this directly on vault? Seems to me this should be Vault not Temporal functionality. I thought there might be some separation of access such that whoever has access to the worker charm didn't necessarily have access to vault to change credentials - this seems to go against that.

[kelkawi-a]

This is a case of separation of access v practicality. We had a similar discussion for the Temporal charm when adding the OpenFGA relation in this PR, where we added utility actions for interacting with the OpenFGA store through the Temporal charm.

There is a way of doing it directly through the vault snap, but not through actions on the Vault charm. I opted to go down a similar path as the Temporal charm and add some utility methods that allow users to easily add secrets to Vault through the Temporal worker charm.

[gtato]

if this is required at the moment to make the charm functional, ok, but i agree with Amber that this defies a bit the point of Valut. If the same party can read and write secrets in/from Vault, there is practically no difference between using vault or an environment variable. If we have a centralized vault, we'd assume that different people add secrets potentially via the UI.

[onurmus]

I left some small comments but none of them is a blocker. Great work!

[gtato]

Left some comments, nothing blocking.

[gtato]

what if we called it environment.yaml (since not all envs are secrets, but all secrets are envs)
what if we put env as root:

env:
    plain (or default):
     - key1: value1
    juju:
     ...
    vault:
     ...

or a more k8s-ish syntax

env: 
   - name: key1
     value: val1
   - name: key2
     valueFromSecret:
        type: juju
        id: secret-id
        key: key2 
   - name: key3
     valueFromSecret:
        type: vault
        id: secret-path
        key: key3      

or a more compact form of this:

env: 
   - name: key1
     value: val1
   - name: key2
     valueFromSecret: juju@secret-id:key2
   - name: key3
     valueFromSecret: vault@secret-path:key3 

[kelkawi-a]

Agreed. I went with the low impact change, as I'd like to avoid having the config look k8s-ish, and the compact form might have readability issues.

[gtato]

question: the key here matches both the key in the secret and the name of the environment variable right? I think there might be cases when that could be annoying because as a developer you are obliged to use an env that you might not know from the beginning

[kelkawi-a]

Not sure I understand but yes the key in the secret must match the key that the user specified in their code. What would be an alternative?

[gtato]

this is a bit related to the convo Amber touched on. People who set secrets should in general be different from the ones who write the code. So in the code you know you need some environment variable, but you should be able to call it whatever you want (say MY_ENV). Similarly, the person who writes the secret, can put a whatever key (say my_secret). Only at the moment of the deployment, you should say: load my_secret value into MY_ENV. We should not assume that the programmer knows that the key is my_secret and do os.getenv('my_secret') from the beginning.

[kelkawi-a]

Good point. So for juju and Vault, I guess we will need to add an optional key of what the env variable name should be? Something like:

juju:
   - secret-id: <secret_id>
     src-key: key
     dest-key: MY_ENV

Or it could be (I like this more):

juju:
   - secret-id: <secret_id>
     key-name: MY_ENV
     from-key: key

Any suggestions?

[gtato]

that is fine, but then i think you would need to keep the env consistent:

env:
   - name: MY_ENV1
     value: bla bla
juju:
   - name: MY_ENV2
     secret-id: secret-id
     key: key2
vault:
   - name: MY_ENV3
     secret-path: secret-path
     key: key3

[gtato]

if this is required at the moment to make the charm functional, ok, but i agree with Amber that this defies a bit the point of Valut. If the same party can read and write secrets in/from Vault, there is practically no difference between using vault or an environment variable. If we have a centralized vault, we'd assume that different people add secrets potentially via the UI.