Data encryption - PostgreSQL Flexible Server

[nraooptum]

We have a policy / compliance requirement to use customer managed keys. I see that the PostgreSQL Flexible Server resource supports geoBackupKeyUri and primaryKeyUri which is great, but I don't see a way to generate those CMKs with ASOPv2. In terraform, we created one via azurerm_key_vault_key and gave the user-assigned managed identity access to it and nabbed the key URL from that. How can we achieve this with ASOP v2?

[nraooptum]

Also, it seems we cannot supply a value from a config map for the key URIs either.

[theunrepentantgeek]

We don't yet support creating Keys within Keyvaults, we're tracking that in #3188. I'll tag that for consideration when we triage things for our v2.11 release.

Enabling configuration for the Key URIs should be straightforward; I'll check into that and report back.

[nraooptum]

I was able to work around that by deploying a custom (in-house) operator to augment the CMK functionality. I'll look at getting it over to ASO as well when I'm not so strapped for time. As for adding the missing config map support, looks like it's just a matter of adding $importConfigMapMode: optional to those fields in azure-arm.yaml?

[theunrepentantgeek]

To make them configurable, yes - and I have that change in a working branch already. The most tedious bit was confirming the generated code is correct.

However, I'm wondering if those are actually ARM URLs, not arbitrary ones - which would instead involve tagging them with $isARMResource: true and they'd change from a string to a KnownResourceReference. Then, they could be configured with an ARM URL to reference a key in Azure, or with details of a Key within the cluster (once #3188 is done).

[theunrepentantgeek]

We've confirmed that while these keys are URIs, they're not Azure Resource URIs - they come from the KeyVault dataplane API. PR #4311 will be merged shortly and released in the next version of ASO.

[nraooptum]

Awesome -- looking forward to testing it. Yeah, from our tenants I can see it uses *.vault.azure.net for public cloud and *.vault.usgovcloudapi.net for azure gov cloud for example.

[nraooptum]

@theunrepentantgeek is there anywhere I can go to get a nightly / early release version of the ASOP image? I would love to test it and give you my feedback. Plus, it would simplify quite a bit of logic in my custom operator.

EDIT: https://github.com/Azure/azure-service-operator/releases/tag/experimental

[nraooptum]

It's probably something I'm doing, but I'm getting:

failed to add initial resource state: transforming resource neil-cdk8s-test to ARM: looking up configmap for property GeoBackupKeyURI: couldn't find resolved genruntime.ConfigMapReference Name "cmk-neil-cdk8s-test-backup-cmk-cm", Key: "uri"

Barring the naming convention which I have to fix, the config map itself definitely exists in the same namespace as the FlexibleServer resource, the only difference is this config map was generated by another pipeline instead of ASO:

[matthchr]

You may have found a bug, related to preview versions.

The root cause is (I suspect):

1. The storage version is 20221201, which only has PrimaryKeyURI as a field, because GeoBackupKeyURI doesn't exist in that API version.
2. The ConfigMap resolution code in the ASO reconciler uses reflection to walk the object being reconciled and discover all of the configmap references that it has, which it then goes and looks up/discovers. My guess is that for some reason it's not discovering the genruntime.ConfigMapReference ref that should be in the propetyBag of the storage type. I don't know why this is as based on a quick read of the code I think it should work
3. Since it didn't discover the ConfigMapReference, it never read it from the Kubernetes API and that's what triggers the error you're seeing.

If the above is true, we should be able to reproduce this easily too.

Did you also try to use the PrimaryKeyURL field? If I'm correct and the above is the problem it will work but GeoBackupKeyURI won't.

[nraooptum]

You are right, if I include just the primary key URI field, it gets past that error and communicates with ARM, which (correctly) errors out:

{
          "name": "ee57bb94-9b7f-4dfb-805d-fd81cec64195",
          "status": "Failed",
          "startTime": "2024-10-02T20:21:46.937Z",
          "error": {
            "code": "RequireTwoEncryptionKeysAndIdentities",
            "message": "For Geo redundant backup enabled server 'psql-test-neil-cdk8s-test-dev-centralus' with Customer Managed Key encryption, need to specify 2 keys and 2 identities. See https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-data-encryption#using-data-encryption-with-customer-managed-key-cmk-and-geo-redundant-business-continuity-features-such-as-replicas-and-geo-redundant-backup"
          }
        }

[matthchr]

I filed a bug tracking this. I'm doing related work in a related area on #2555 as well. Plan is to get this fixed before 2.10. I had realized it was an issue for the work I was doing on #2555 (which is about export) but hadn't put 2+2 together that it was already impacting existing behaviors as well for import.

Thanks for the report and the patience!

[nraooptum]

Thanks Matt! I'd be happy to test out a potential fix when it's available. I've got an internal demo / presentation coming up in two weeks to present my prototyping with ASOP and CDK8s as a potential organizational shift to these technologies, so I'm eager to get this demo wrapped up and working.

[matthchr]

We're about to do a release but you can try this out in the latest experimental in the meantime.

Though actually it looks like it's hitting a build issue so we'll need to fix that first... will try to update here once the updated experimental is out.

[matthchr]

There should be an ASO 2.10 release soon, but you can try this now with the experimental release if you like.

[nraooptum]

Thanks Matt! I'll give this a go sometime this week and report back.