WebhookServer sslcontext and certificate file refreshing

[stevenolen]

I've been poking around the codebase with regard to the WebhookServer implementation to find an answer to a musing, but have fallen a bit short.

I'm looking to set up a webhookserver using cert-manager provided certificates. My understanding is that this is pretty easy -- create the Certificate resource, mount the referenced Secret into the operator pod and pass the file location on init:

@kopf.on.startup()
def configure(settings: kopf.OperatorSettings, **_):
    # assumes the secret is mounted at `/certs`
    settings.admission.server = kopf.WebhookServer(port=443, certfile='/certs/tls.crt', pkeyfile='/certs/tls.key')

Diving down into WebhookServer a bit, I found _build_ssl() which creates an ssl.SSLContext via this call, and from here I have a question: what happens while the operator is running if cert-manager re-issues this certificate? I'm not seeing any obvious place where this context is refreshed or reloaded. Perhaps at the moment if the operator continues to run until the certificate expires, it will stop working?

Thanks!

[nolar]

Yes, it will stop working one way or another (I did not test this scenario). There is no reloading of certificates after their expiration — for the sake of simplicity, mainly.

One way would be to restart the whole operator. Another way would be to inherit from the provided webhookserver class (or write a new one) and reload the certificates on desired triggers/conditions. The design of those classes is intentionally extensible.

[stevenolen]

That makes a ton of sense. I was pretty surprised to find that the controller-runtime (which underpins operators and controllers created from kubebuilder/operator-sdk) actually have a filewatcher built in for exactly this purpose!

Thanks for the quick response! kopf is a really wonderfully built framework!

[nolar]

One little update, which might be not obvious: a webhook server can yield not 1, but 2, 3, 4+ ClientConfigs over its lifetime. Every time it yields a config, the config is applied to the managed webhook configurations. Typically, it only yields 1 and then sleeps forever. But you can take a look at the "tunnelling servers" (in the same file), which wrap an existing WebhookServer and yield from it, potentially many times. As such, you can make the logic with expiration time tracking that will cancel the "current" wrapped webhook server and start a new one with the new cert, but yielding from them all without leaving the wrapper's routine. Semantially, something like:

def __call__(…):
    yield from WebhookServer(cert='cert1.pem')
    yield from WebhookServer(cert='cert2.pem')
    …

[nolar]

It would be good to have this file watching capability in Kopf (with proper tests), but I am quite sure I do not have enough time & energy to implement that (I'm now finishing a 2-year-old refactoring of the core & tests, sloooooowlllyyyy).