RFC: Trustee support High Availability

[huoqifeng]

There are two kinds of resources are saved in trustee today:

• secrets/resources used by CoCo Pod
• policies used by Trustee/KBS itself

I think KBS supported HA already if the secrets/resources are saved in KMS and if trustee get the resource from KMS plugin. But if the resources are saved in KBS directly, it's not HA yet. We'd better have an approach to enable HA if user select to save resources in KBS directly.

For policy files, it's not HA yet because a damaged/recreated trustee Pod will cause all the policies files missing.

Maybe, there are two options here:

1. Use persistent storage to save all the resources and policies file. The storage should. be encrypted
2. Use ConfigMap in k8s to save all the resources and policies file.

I created this issue to track and discussion the approach.
@fitzthum @Xynnn007 @mkulke

[fitzthum]

I am in favor of this. @mkulke has been a big proponent.

One thing to keep in mind is that we also have this issue confidential-containers/trustee-operator#70 where we are trying to figure out how to deploy Trustee in CoCo.

This conflicts slightly with some of the k8s-based approaches to handling resource, policies, etc. since we don't want to Trust the control plane.

We might just need to support two different paradigms.

[Xynnn007]

Let's try to decouple this problem. On the one hand, it is the high availability of trustee storage secrets and policies, and on the other hand, it is the data security of CoCo deployment.

It should be emphasized that my discussion does not assume that Trustee is deployed through CoCo, ordinary K8S is also fine, as long as it is in the user's trusted domain. Therefore, we do not have a strong need to encrypt a Trustee resource that has been deployed in the user's trusted domain.

First, let’s talk about the high availability of trustee. Store the secret in the KMS plug-in, or put it in a Volume that is highly available, and then bind it to the KBS container to achieve high availability. As for the policies, the same strategy can be used, but Trustee semantics supports writing, so we have to additionally deal with the issue of write consistency. I believe this problem has been solved in the traditional k8s ecosystem (for example, using file systems such as Ceph as storage plug-ins) and not new for CoCo.

Secondly, about data confidentiality of CoCo’s deployment. The principle of persistent encrypted data security should be similar to kata-containers/kata-containers#9999, which further supports transparent encryption of persistent Volumes.