[V2] Program controller and file server for program objects

[rquitales]

Proposed changes

This PR enables a Workspace pod to fetch a Program object in a similar fashion to how it fetches Flux artifacts. We do this by exposing a HTTP server that serves tarballs of a fully formed Pulumi.yaml file, incorporating the requested Program spec. Unlike the approach with the Flux source-controller, I've opted not to create/store these tar files in local ephemeral storage in the controller pod. Instead, when the artifact URL is accessed, the file server will fetch the requested Program resource from the Kubernetes API server, and wrap it up in a Pulumi.yaml file and tarred. This approach ensures that the most recent Program spec is always served, and it also greatly simplifies storage since we do not need to create a local duplicate of these Program objects. Since the source of truth is always on cluster, we do not need to continuously reconcile and generate new artifacts for new generations of Programs. Should we choose to change this implementation in the future, the current strategy using the status field to convey the URL should make it easy to do so.

• Add status field to the Program resource to advertise a downloadable URL for the program
• Scaffold a program-controller to reconcile the status/URL
• Create a simple file server to serve the fully-formed Pulumi.yaml from a Program URL
• Update deployment manifests to expose the file server
• Additional unit tests
• Rebase PR to take in test changes
• Integrate with stack-controller

Related issues (optional)

Closes: #651

[codecov]

Codecov Report

Attention: Patch coverage is 26.97674% with 157 lines in your changes missing coverage. Please review.

Project coverage is 33.11%. Comparing base (af4964f) to head (c07c641).
Report is 1 commits behind head on v2.

Files with missing lines	Patch %	Lines
...r/internal/controller/pulumi/program_controller.go	35.53%	69 Missing and 9 partials ⚠️
operator/cmd/main.go	18.30%	56 Missing and 2 partials ⚠️
...tor/internal/controller/pulumi/stack_controller.go	0.00%	17 Missing ⚠️
...ator/internal/controller/pulumi/metrics_program.go	33.33%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               v2     #673      +/-   ##
==========================================
- Coverage   33.25%   33.11%   -0.15%     
==========================================
  Files          25       27       +2     
  Lines        3951     4161     +210     
==========================================
+ Hits         1314     1378      +64     
- Misses       2475     2610     +135     
- Partials      162      173      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[EronWright]

I don't understand; multiple stacks could use the same program, no?

[EronWright]

I still think this field should be removed because a) it is redundant given the observedGeneration, and b) it uses the term "sync" which has a particular (different) meaning in PKO.

[rquitales]

• field should be removed: I'm opposed to this more so because we can have multiple reconciliation on the same generation. Take the example where a Program object has been reconciled and the status is written to it. If the controller pod restarts, there might be a possibility that the hostname has changed (though very unlikely). When the controller pod restarts, it will reconcile all objects it should watch for, and thus, the URL stored in status will be updated. It would be helpful for debugging purposes to know when the URL was generated.
• the term "sync" which has a particular (different) meaning in PKO.: yes, a different name for this field, should we ultimately decide to keep it, should be used!

[rquitales]

I've removed this field for now, to merge this PR in eagerly. We can rethink how we want to handle this and add this in the future if we choose to.

[EronWright]

Looks great, just left a couple of comments.

[blampe]

Looks good, mostly just some nits but nothing blocking.

[blampe]

Nice!