Bindable Secrets Resource

[tcdowney]

This RFC proposes adding a secret resource to the V3 Cloud Foundry API, similar in design and purpose to the Kubernetes Secret resource. Cloud Foundry Secrets will store arbitrary data in CredHub inject it into app containers via tmpfs-mounted files at a given path, using the same mechanism as RFC 0030 - Add Support for File based Service Binding Information.

---

Readable Version

[Gerg]

cc @cloudfoundry/wg-app-runtime-interfaces-capi-approvers @cloudfoundry/wg-app-runtime-interfaces-cli-approvers @cloudfoundry/wg-app-runtime-platform-diego-approvers

[sethboyles]

question: Any thoughts about how secrets would be managed in app/space manifests?

[tcdowney]

Good call out. I think they should work like services with an array of secret names which will create secret bindings, I'll add a section on this.

Whether or not secrets themselves should be defined in a v2 space manifest is in another question. 🤔

[beyhan]

Whether or not secrets themselves should be defined in a v2 space manifest is in another question.

@tcdowney what do you mean by this?

[tcdowney]

@Gerg is working on a draft proposal of a "v2" version of the CF App manifest that is more declarative and includes the ability to configure other space scoped resources. This is still WIP, but he has a version of it here:
https://github.com/Gerg/community/blob/rfc_v2-app-manifests/toc/rfc/rfc-draft-v2-app-manifests.md#add-secrets

[tcdowney]

@winkingturtle-vmw @ameowlia

Are there any paths we'd need to prevent here that would interfere with Diego or otherwise be bad? Allow app devs to drop arbitrary files anywhere could be bad.

[geofffranks]

Can we have the secret fail to mount + fail to start the app, if the mount path is already an existing directory on the container filesystem?

These mounts would replace the directory structure they're mounted on top of, rather than be additive, right? If so, this should prevent most problems related to masking system directories/files, or accidentally masking app code or other required resources in the app container.

[ameowlia]

❓ Could we make a dir where the secrets always go to? Where users could make nested dirs if they want. But this way we could limit exactly where they are putting things and not need to worry about it?

[Gerg]

The downside of limiting secret paths to a single directory is for off-the-shelf apps that are picky about where their secret files should be (there is additional discussion around this in other threads).

[Samze]

Thought Maybe its too late, but do we need both ServiceBindingFiles and Secret fields here? Can these both be considered just two lists of files to mount where the value can be either explicit or de-referenced from Credhub, and Files for the purpose of ServiceBinding mounting just happen to share the same mount path SERVICE_BINDING_ROOT

[tcdowney]

Yeah, I agree.

@beyhan do you have thoughts on how to make this cleaner? How far along is the Service Binding Files work for us to potentially generalize the BBS interface?

[beyhan]

I think my comment #994 (comment) is also relevant here.

[tcdowney]

I know this may be too late, but is the name of the field ServiceBindingFiles set in stone? I'm wondering if we could make it more generic -- even VolumeMountedFiles or something. I just don't know how far along the file-based service binding implementation is at this point.

cc @Gerg

[Gerg]

Ideally I would have noticed this in the file-based binding RFC, but this does feel like CF-level concepts leaking into BBS's domain model. Historically, BBS has tried to separate itself from CF concepts (e.g. app vs LRP).

[beyhan]

Ideally I would have noticed this in the file-based binding RFC, but this does feel like CF-level concepts leaking into BBS's domain model. Historically, BBS has tried to separate itself from CF concepts (e.g. app vs LRP).

I think at the stage which we are with the implementation it should be possible to change ServiceBindingFiles to VolumeMountedFiles. @stephanme, @PlamenDoychev any thoughts on this?

[beyhan]

I opened #1008 to address the point about leaking CF-level concepts into BBS's domain model.

[tcdowney]

Thanks @beyhan , I will update this RFC to reflect those changes.

[Samze]

question Do you think its likely people will want to create Secrets that contain service info and mount them at $SERVICE_BINDING_ROOT so libraries like https://github.com/spring-cloud/spring-cloud-bindings can read them? Would anything in this design stop that? I suppose the developer would have to know what the $SERVICE_BINDING_ROOT value is and explicit bind to that mount path? Or alternatively, do you think for services users should really be using UPS?

[stephanme]

RFC 0030 - Add Support for File based Service Binding Information requires to configure an app feature flag and then switches all service bindings from VCAP_SERVICES to files under $SERVICE_BINDING_ROOT.

I could imagine that secrets and secret bindings could allow a smoother migration path from VCAP_SERVICES to file based service bindings by allowing both mechanisms in parallel.

If secret bindings are allowed to mount under $SERVICE_BINDING_ROOT, a prioritisation is needed between service bindings and secret bindings.

[tcdowney]

Hmm, I can see arguments for both. My gut tells me that service bindings should take priority, but I also think it is most flexible for devs if secrets take precedence.

For example, imagine you have a MySQL instance hosted on one instance of CF that creates publicly reachable service instances (exposed via TCP routes) and you want to bind to these from apps on other Cloud Foundry installations. One workflow would be to make a Service Key on the "services CF" and then create UPSIs on the ones where your apps are running based on that key.

Problem is an UPSI does not have the correct type that helper library might expect (and not all libraries look at tags). If you could use a secret for this then you could explicitly set the keys to what you want to as a dev. But semantically the UPSI still makes more sense here -- so maybe that interface just needs to be more flexible.

Thoughts @Samze @stephanme @Gerg ?

[Gerg]

We have a kinda similar situation with environment variables. We intentionally restrict users from setting environment variables that conflict with VCAP_SERVICES and friends, but there can be collisions between global environment variables and app-level ones.

[beyhan]

I think having secret bindings under$SERVICE_BINDING_ROOT will make also the interface between CC and Diego simpler. Most probably we will need to introduce only the ServiceBindingFiles: as new structure. Also, activation of the filed based approach via the app feature flag can be reused. Additionally, there is also a validation and setup code for the $SERVICE_BINDING_ROOT which makes sure that we have enough disk space for alle the container mounts on the Diego VMs as specified in

community/toc/rfc/rfc-0030-add-support-for-file-based-service-binding.md

Line 45 in 909be76

Additionally, the application environment is stored in the `CCDB` and `BBS DB` that is why we should define a limit for the size of it, which makes it possible to be stored in the according DBs and doesn’t impact the performance of the communication between Cloud Controller and the Diego API. That is why this RFC suggests a limit of `1MB`, which is roughly ten times higher than the current one of 130KB. This is subject for evaluation during the implation of this RFC.

. If we introduce another mount point for the secret bindings we have to define limits and implement validation for that. Most probably having the $SERVICE_BINDING_ROOT as root for the secret bindings will increases the adoption efforts of the mentioned commercial off-the-shelf software in https://github.com/tcdowney/community/blob/c767d5e0c61c04f8a5be1a45fb285b942794a7ad/toc/rfc/rfc-draft-bindable-secrets.md?plain=1#L21.

[blgm]

We acknowledge that there is some overlap between bindable secrets and user-provided service instances ("UPSIs"); and also to some extent "managed" (broker provided) services ("MSIs"), particularly when/if RFC 0030 - Add Support for File based Service Binding Information is implemented too. I can foresee folks struggling to differentiate: should I use a bindable secret or an UPSI?

If we do this right, we could simplify the platform - for example by making UPSIs and MSI subtypes of bindable secrets, and deprecating UPSIs. If we do this wrong then we could add a forever confusion about what's the right choice, and then have to document and train folks out of the mess.

Perhaps this is too radical, but I'd be interested on folk's thoughts about:

• deprecating UPSIs and changing the commands to recommended a bindable secret instead (but not breaking anything yet)
• teaching that MSIs are a special type of bindable secrets rather than a different concept

[beyhan]

• deprecating UPSIs and changing the commands to recommended a bindable secret instead (but not breaking anything yet)

My thoughts on this. I think we can have a chance here if we could offer the migration of UPSis to bindable secrets without requiring any changes in the application code. That means, the libraries parsing service binding information need to be adjusted and do the heavy lifting. I'm not sure whether this means also, that bindable secrets need to support not only file based approach for the service binding but also environment based because the move from UPSis to bindable secrets will mean also to opt-in into the file based service binding approach.

[stephanme]

Deprecating UPSIs and replacing it with bindable secrets avoids the confusion of two solutions for a very similar problem.

I don't think that we can change the existing functionality for UPSis with VCAP_SERVICES. But if a deprecation shall be a bit more serious than just the word "deprecated" in the docs, we could take out UPSis from RFC0030 File-based Service Bindings.

Existing CF apps (using VCAP_SERVICES) need to be changed anyway when switching to file-based bindings, be it RFC0030 service bindings or bindable secrets.

[tcdowney]

I don't think we should deprecate UPSI's. They are still semantically distinct from a generic Secret -- they provide a way to provision services off platform (either on a separate Cloud Foundry instance or via some other mechanisms) and provide service information to apps through similar a similar UX as a fully managed service.

I think although Secrets could be used to accomplish something similar, we would be going against over a decade of ingrained UX for CF developers. I also think supporting file-based service bindings for UPSIs is important for service offerings (both older CF oriented ones and newer K8s ones) that may not conform exactly to what the various service binding loading application libraries expect. If a specific key is slightly different, for example, an UPSI can be used to massage its presentation to what the library would need.

I guess in short, there are actually some use-cases that file-based UPSIs would provide that I am interested in leveraging and I wouldn't want to gate them behind a non-implemented (and not yet approved) Secrets proposal. 😅

[Gerg]

I agree with Tim.

Similarly, there is "duplication" between cf cups and cf set-env: letting you set environment variables in app containers. You could theoretically achieve logical "service bindings" with cf set-env, and you could configure your app using UPSIs. However, we still keep both of those around because they serve semantically different purposes, are intended for different user workflows, and affect different segments of the app container's environment variables.

They may be implemented with the same mechanics under the hood, but that doesn't mean we need to expose that to our users.

[blgm]

There's not a lot in this doc yet about permissions. I infer that an app dev with access to an App will be able to create and bind secrets, and that makes sense, but what if we go further? In the past folks have talked about an Independent Service Manger (ISM) that's a bit like a service broker, but is independent of the platform and might not implement OSBAPI. What if there was a way to give an off-platform service the ability to create secrets, but have no access to Apps? For example if I'm connecting to a MySQL on AWS, I might use AWS Credential Manager to rotate my password every few days. What if a Lambda on AWS could be triggered by this change, and update a secret in CF? We would not want that Lambda to have access to Apps, or other secrets on CF. But it would be ok if could update its own secret. Equally we might allow LetsEncrypt to update a secret that's a certificate. Maybe this is out of scope for now, but it would be nice to make sure we don't block anything like that in the future.

[tcdowney]

Yeah, I think this is out of scope for now. I will add a section around permissions mentioning that Secrets can be created and managed by users with the SpaceDeveloper role since this will treat them the same as other sensitive resources like app environment variables and service binding credentials.

[joaopapereira]

How do we create this secret via CLI?
will we have a file with the following yaml

application.yaml: |-
  ---
  application:
    test
secret.txt: my-secret

?

[joaopapereira]

If we do not have a different type right now is it worth it to add this flag?

[tcdowney]

Yeah, we don't need it for now. I'll update this and remove the flag.

[joaopapereira]

Is there going to be an endpoint to get the full secret "object"?
Is there going to be a CLI command to retrieve the secret?

[tcdowney]

There probably should be a CLI command, but as it stands having a separate endpoint for the non-sensitive bits of the Secret and the Secret value has parity with how env vars and service binding credentials work. This is mostly necessary because of how permissions work in Cloud Controller (we want some roles to be able to see the existence of a Secret / Binding to an app, but not its contents).

The CLI could call both endpoints for SpaceDeveloper users and show more details if that's desired.

[ameowlia]

❓ Are we worried about different secrets writing to the same place?

1. I make a secret named "meow-secret" with value {"meow.txt": "meow"} and bind it to my app.
2. I make a secret named "also-meow-secret" with value{"meow.txt": "woof"} and bind it to my app.
3. What happens here? Is there an error? Is it non-determanistic?

[tcdowney]

❓ Are we worried about different secrets writing to the same place?

1. I make a secret named "meow-secret" with value {"meow.txt": "meow"} and bind it to my app.
2. I make a secret named "also-meow-secret" with value{"meow.txt": "woof"} and bind it to my app.
3. What happens here? Is there an error? Is it non-determanistic?

One option is to make mount_path unique per app in the secret_binding, but I feel there are valid reasons to allow multiple secrets to be bound to the same mount_path dir. In my opinion it's fine if this is non-deterministic since the space dev is in control of what happens here (they chose to mount them both to the same path).

[geofffranks]

❓ Are we worried about different secrets writing to the same place?

1. I make a secret named "meow-secret" with value {"meow.txt": "meow"} and bind it to my app.
2. I make a secret named "also-meow-secret" with value{"meow.txt": "woof"} and bind it to my app.
3. What happens here? Is there an error? Is it non-determanistic?

One option is to make mount_path unique per app in the secret_binding, but I feel there are valid reasons to allow multiple secrets to be bound to the same mount_path dir. In my opinion it's fine if this is non-deterministic since the space dev is in control of what happens here (they chose to mount them both to the same path).

If we go the route of preventing mount path from mounting on top of an existing directory, this sorts itself out with no additional logic at the CAPI layer. Otherwise it could get expensive to check that the credentials inside a single secret do not conflict with any of the mount paths/credentials of any of the other secrets that are bound to any of the apps bound to the secret being updated.

[philippthun]

❓ Are we worried about different secrets writing to the same place?

1. I make a secret named "meow-secret" with value {"meow.txt": "meow"} and bind it to my app.
2. I make a secret named "also-meow-secret" with value{"meow.txt": "woof"} and bind it to my app.
3. What happens here? Is there an error? Is it non-determanistic?

One option is to make mount_path unique per app in the secret_binding, but I feel there are valid reasons to allow multiple secrets to be bound to the same mount_path dir. In my opinion it's fine if this is non-deterministic since the space dev is in control of what happens here (they chose to mount them both to the same path).

At some point CAPI builds this "file structure" passed to Diego, i.e. it iterates over all the secrets and adds them to the VolumeMountedFiles array. Here conflicts will be easily detectable and should - from my point of view - result in an error.

[a-b]

question: Are we aware of any encoding constraints? For example, are we expecting binary secret data encoded to base64, or is this not required?

[tcdowney]

question: Are we aware of any encoding constraints? For example, are we expecting binary secret data encoded to base64, or is this not required?

This is an interesting point. I hadn't considered those types of secrets, but that would be important to fully support COTS / k8s-deployed apps that have non-unicode secrets. Maybe I should update this to look more like the K8s implementation (stored and read as base64, but users can provide data: or stringData: when creating one). 🤔

[beyhan]

@tcdowney, @Gerg I see that there are still open discussions and pending updates to this RFC. Pls let me know when you think that it is ready for final review and then staring the FCP.