feat(deployment): move service account passwords to autogenerated secrets for production

[theosanderson]

Uses service account passwords from autogenerated secrets. Adds special E2E configuration to allow the pipeline password to be hardcoded, as we need to pretend to be the pipeline from the E2E test and the E2E tester doesn't have access to the autogenerated sequences.

https://move-to-secrets.loculus.org/ebola-zaire/search

[corneliusroemer]

This is good for production (pathoplexus) but I would prefer if we don't do this on loculus as it makes pipeline testing much harder.

Spinning up a local full deployment isn't straightforward at the moment so testing against test deployments is very beneficial.

Rather than making it an E2E vs rest distinction we could make it a production vs development one, or are you worried the production is insufficiently tested as a result?

[theosanderson]

@corneliusroemer: the default values.yaml needs to be a secure environment for production. We can add additional values files for development (as we do already for E2E here). If the --dev flag in deploy.py used those same values (i.e. hardcoded these passwords) would that work for you?

[theosanderson]

Yes, IMO preview deployments should be as similar to the real thing as we can get them. If you want secondary test deployments through ArgoCD we could try to make those, but the intent of preview deployments is to preview how something will work in production.

[corneliusroemer]

Is it possible to make it easy to get a hard coded argocd deployment by toggling a field in values.yaml, like we do with disable preprocessing etc? That would be fine, then I can't test again main anymore but make a quick test deployment.

We still have superuser with fixed passwords in all preview deployments btw 😀

deploy.py isn't enough as local deployments aren't complete and also generally more cumbersome than argocd preview (lacking backend and website arm images)

[theosanderson]

It can't really be a single field, but all you'd need to do would be to merge the values_dev.yaml into values_preview_server.yaml so ~10 secs in VS code

[theosanderson]

FWIW deploy.py can be used to make a full deployment - that's how the E2E tests work
Sorry I misread the point about ARM images

[corneliusroemer]

I'm testing the suggested merge here: #1746

Unfortunately spruce reorders the values.yaml making the diff appear big. Maybe a sed script would be more light touch.

A boolean would be cool but I can see it's not so easy. Maybe just changing two lines in question might be easier than going full merge.

[theosanderson]

All you should need is this: https://github.com/loculus-project/loculus/pull/1747/files