V2 #725
Merged
V2 #725
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
<!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md)
for Pulumi's contribution guidelines.
Help us merge your changes more quickly by adding more details such
as labels, milestones, and reviewers.-->
### Proposed changes
**Epic Link**:
#606
**Demo Video Link**:
https://pulumi.slack.com/archives/C07DQSV84DC/p1722636430008649
Implements an agent consisting of two commands:
- `init` - fetches a flux source into the given directory, intended for
use in an init container.
- `serve` - starts an RPC server providing an automation API to perform
stack updates over the given workspace.
### Overview
The RPC server assumes that the project source code has been checked out
to a local working directory, called the "workspace" directory. This
generally corresponds to a sub-directory within a git repository, e.g.
[examples/random-yaml](https://github.com/pulumi/examples/tree/master/random-yaml).
At startup, the server opens the workspace using
[auto.NewLocalWorkspace](https://github.com/pulumi/pulumi/blob/5651750bb254f73da5ef0fa503818c5a38755ea8/sdk/go/auto/local_workspace.go#L848).
All RPC operations are applied to this same workspace, usually
one-at-a-time. Some operations cause state changes, e.g. stack
selection, that may affect subsequent operations. Some operations
produce `PreconditionFailed` if a stack hasn't been selected.
At startup, the server optionally runs `pulumi install` to install
dependencies and plugins for the project, based on
pulumi/pulumi#16782. Note that PKOv1 has some
code to install plugins, now believed to be obsolete (see
[discussion](pulumi/pulumi#16782 (comment))).
The supported operations are:
- `WhoAmI` - returns current user info.
- `Install` - runs `pulumi install` in the workspace.
- `SelectStack` - select (and optionally create) a stack, for use in
subsequent operations.
- `Info` - a summary of the current stack.
- `SetAllConfig` - set multiple configuration values on the current
stack, based on literals, environment variables, and file references. It
is expected that the server's pod would have ConfigMaps and Secrets
mounted accordingly.
- `Preview` runs the preview operation for the current stack.
- `Up` runs the up operation for the current stack.
- `Destroy` runs the destroy operation for the current stack.
- `Refresh` runs the refresh operation for the current stack.
The deployment operations have streaming responses, consisting of a
series of engine events and a final result.
The agent uses zap for logging, because it supports structured logging,
implements `io.Writer` to capture Pulumi console output, and integrates
well with grpc-go.
### Follow-ups
- [x] Write RPC server tests
- [ ] Rename 'init' to 'fetch' for clarity
- [ ] lock the workspace during an operation? Or rely on locking within
the Pulumi CLI?
### Related issues (optional)
<!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes
#1234'.
Or link to full URLs to issues or pull requests in other GitHub
repositories. -->
Closes #610 #611
<!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md)
for Pulumi's contribution guidelines.
Help us merge your changes more quickly by adding more details such
as labels, milestones, and reviewers.-->
### Proposed changes
<!--Give us a brief description of what you've done and what it solves.
-->
This PR implements the `auto.pulumi.com` API Group, including the
`Workspace` and `Update` types.
Integration tests for the workspace controller are included.
### Example
```yaml
apiVersion: auto.pulumi.com/v1alpha1
kind: Workspace
metadata:
name: random-yaml-1e2fc47
spec:
image: pulumi/pulumi:3.128.0-nonroot
securityProfile: restricted
serviceAccountName: default
git:
url: https://github.com/pulumi/examples.git
revision: 1e2fc471709448f3c9f7a250f28f1eafcde7017b
dir: random-yaml
env:
- name: PULUMI_ACCESS_TOKEN
valueFrom:
secretKeyRef:
name: pulumi-api-secret
key: accessToken
resources:
requests:
cpu: 1
memory: 512Mi
limits:
cpu: 1
memory: 512Mi
# various extension points shown here.
# - custom pod labels
# - pod tolerations
# - extra init container(s)
# - extra volume(s) and volume mounts onto the 'pulumi' container
podTemplate:
metadata:
labels:
example.com/mylabel: bar
spec:
terminationGracePeriodSeconds: 3600
tolerations:
- key: "example.com/foo"
operator: "Exists"
effect: "NoSchedule"
initContainers:
- name: extra
image: busybox
command: ["sh", "-c", "echo 'Hello, extra init container!'"]
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_BIND_SERVICE
drop:
- ALL
volumeMounts:
- name: share
mountPath: /share
containers:
- name: pulumi
volumeMounts:
- name: secret-volume
mountPath: /etc/secret-volume
readOnly: true
volumes:
- name: secret-volume
secret:
secretName: test-secret
```
### Specific Changes
- (agent) git source support
- (operator) scaffolding
- (operator) workspace api and controller and tests
- (operator) update api and controller
- (experimental) configuration block in `Environment` spec
- (experimental) dockerfile for non-root pu/pu
### Related issues (optional)
<!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes
#1234'.
Or link to full URLs to issues or pull requests in other GitHub
repositories. -->
Closes #619
<!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md)
for Pulumi's contribution guidelines.
Help us merge your changes more quickly by adding more details such
as labels, milestones, and reviewers.-->
### Proposed changes
<!--Give us a brief description of what you've done and what it solves.
-->
This PR re-implements the Stack controller in terms of the low-level
`Workspace` and `Update` subordinate objects.
Follow-ups:
- git source support w/ polling
- tests for the stack controller
- stack outputs in Stack status (requires an enhancement to the Update
API)
- optimized handling of the stack config in Workspace API
### Related issues (optional)
<!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes
#1234'.
Or link to full URLs to issues or pull requests in other GitHub
repositories. -->
This adds the groundwork for e2e testing the v2 branch. Try it with `make test-e2e`. I haven't enabled an assertion for the random-yaml stack because I wasn't able to see it reliably reconcile. I was seeing GitHub rate limit errors as well as errors from the operator when updating status on the stack.
This adds a `workspaceTemplate` field to the Stack API. Also removes `resources` and `image` from the API, since those can now be specified via `workspaceTemplate` and it's confusing for them to be overridable in two places. We can re-add these as a convenience later when we've thought through precedence and validation for them. Fixes #666
Implements the git source for v2. We currently only support secret refs from the Stack API because other options (fs, env) don't make sense to share between agent and operator. Auth information is passed via env vars to the agent's fetch container. The operator also needs access to these secrets in order to pull for the latest commit. It uses an in-memory `ls-remote` and compares remote refs against's the Stack's ref. We support long and short forms (e.g. `refs/heads/foo` or `foo`). The agent now uses automation API to clone the repo, since it's a natural place for dogfooding. If the init container is restarted, or uses a persistent volume for the checkout, there's an edge case where we won't fast-forward the existing checkout. See pulumi/pulumi#17288. Also adds a `shallow` clone option to the stack API for controlling shallowness. We don't currently perform shallow clones so this is backwards compatible. Fixes #655
<!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md)
for Pulumi's contribution guidelines.
Help us merge your changes more quickly by adding more details such
as labels, milestones, and reviewers.-->
### Proposed changes
Implements a test suite for the new stack controller.
<!--Give us a brief description of what you've done and what it solves.
-->
Closes #654
### Related issues (optional)
<!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes
#1234'.
Or link to full URLs to issues or pull requests in other GitHub
repositories. -->
Stack CRDs that include `envRefs` will currently thrash the workspace's underlying StatefulSet because map order is undefined. This sorts the envRef keys ahead of time to keep the StatefulSet stable. As a result we're also able to turn on the E2E test in CI since it should now succeed reliably.
### Proposed changes This PR enables a Workspace pod to fetch a Program object in a similar fashion to how it fetches Flux artifacts. We do this by exposing a HTTP server that serves tarballs of a fully formed `Pulumi.yaml` file, incorporating the requested Program spec. Unlike the approach with the Flux source-controller, I've opted **not to** create/store these tar files in local ephemeral storage in the controller pod. Instead, when the artifact URL is accessed, the file server will fetch the requested Program resource from the Kubernetes API server, and wrap it up in a `Pulumi.yaml` file and tarred. This approach ensures that the most recent Program spec is always served, and it also greatly simplifies storage since we do not need to create a local duplicate of these Program objects. Since the source of truth is always on cluster, we do not need to continuously reconcile and generate new artifacts for new generations of Programs. Should we choose to change this implementation in the future, the current strategy using the status field to convey the URL should make it easy to do so. - [x] Add status field to the Program resource to advertise a downloadable URL for the program - [x] Scaffold a program-controller to reconcile the status/URL - [x] Create a simple file server to serve the fully-formed Pulumi.yaml from a Program URL - [x] Update deployment manifests to expose the file server - [x] Additional unit tests - [x] Rebase PR to take in test changes - [x] Integrate with stack-controller ### Related issues (optional) Closes: #651
This implements graceful shutdown by propagating interrupts to our child
processes.
Killing a workspace pod in the middle of an update will mark the Update
as failed ("update canceled" error), and subsequent Updates will resume
where it left off as you would expect.
Fixes #607.
<!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md)
for Pulumi's contribution guidelines.
Help us merge your changes more quickly by adding more details such
as labels, milestones, and reviewers.-->
### Proposed changes
Remove any now-obsolete v1 code and fix-up the Makefile. Most of the
top-level build targets work as expected, often by delegating to the
sub-Makefiles.
TODO:
- [ ] CI scripts
Later:
- Deployment-related changes (to take place in a separate PR)
- Top-level tests
#### New Help Output
```
❯ make help
Usage:
make <target>
General
help Display this help.
Development
codegen Generate CRDs and documentation
generate-crdocs Generate API Reference documentation into 'docs/crds/'.
Build
build Build the agent and operator binaries.
build-agent Build the agent binary.
build-operator Build the operator manager binary.
build-image Build the operator image.
push-image Push the operator image.
Deployment
install-crds Install CRDs into the K8s cluster specified in ~/.kube/config.
deploy Deploy controller manager to the K8s cluster specified in ~/.kube/config.
Release
prep Prepare the next release.
Build Dependencies
crdoc Download crdoc locally if necessary. No version check.
```
### Related issues (optional)
<!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes
#1234'.
Or link to full URLs to issues or pull requests in other GitHub
repositories. -->
This returns stack outputs from the agent and records them in a secret. Scrubbed outputs are also stored in the Stack's status, as we do in v1. * `OutputValue` is returned from the agent and contains raw JSON-encoded bytes for the output. * Each `Update` owns a corresponding `-stack-outputs` secret. * The secret includes a `pulumi.com/secrets` annotation with a list of sensitive fields, and the Stack API uses this to scrub outputs for the Stack's status.
<!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md)
for Pulumi's contribution guidelines.
Help us merge your changes more quickly by adding more details such
as labels, milestones, and reviewers.-->
### Proposed changes
<!--Give us a brief description of what you've done and what it solves.
-->
### Related issues (optional)
<!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes
#1234'.
Or link to full URLs to issues or pull requests in other GitHub
repositories. -->
* Consolidate `{operator,agent}/go.mod` under a root `go.mod`.
* Rewrite imports to use v2 path.
* Remove `/test` -- these were still referring to v1 code and can be
revived in a followup if we want to keep any of them.
* Move Dockerfile to repo root and fix image build.
Fixes #687.
---------
Co-authored-by: Eron Wright <[email protected]>
Speed up e2e tests by using the official nonroot image instead of building and loading it locally.
### Proposed changes This update removes the default CRD validation marker from the `workspace.spec.image` field, allowing for dynamic image selection based on the value of `workspace.spec.securityProfile`. When `securityProfile` is set to `baseline`, the image defaults to `pulumi/pulumi:latest`. If `securityProfile` is marked as `restricted`, the system will instead select `pulumi/pulumi:latest-nonroot`. If a user specifies their own image, then that value is used instead. Handling these default settings within the controller is necessary, as CRD validation markers cannot accommodate conditional defaults. Before implementing this feature, additional ginkgo tests were introduced to confirm that the desired behavior is properly achieved. ### Related issues (optional) Closes: #653
<!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md)
for Pulumi's contribution guidelines.
Help us merge your changes more quickly by adding more details such
as labels, milestones, and reviewers.-->
### Proposed changes
<!--Give us a brief description of what you've done and what it solves.
-->
This is a new, simplified deployment app for PKO. It leverages the
kustomization that is maintained in `operator/config/default` to avoid
code duplication. It supports ONLY cluster-wide installation.
PKOv2 is designed to natively support multi-tenancy across namespaces
without needing to deploy separate instances of the operator into each
namespace. Looking back at
#328, we see
that the ability to deploy to multiple namespaces was added to address a
lack of isolation (see pulumi/home#2330). I
opened #690
to track adding support for single-namespace deployment.
### Related issues (optional)
<!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes
#1234'.
Or link to full URLs to issues or pull requests in other GitHub
repositories. -->
<!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md)
for Pulumi's contribution guidelines.
Help us merge your changes more quickly by adding more details such
as labels, milestones, and reviewers.-->
### Proposed changes
<!--Give us a brief description of what you've done and what it solves.
-->
Updates the Helm chart to install PKOv2, as similarly as possible to
`operator/config/default`.
Details:
- adds an aggregation role (view/edit) for the Pulumi API groups
- tweaks the controller's resources such that limits equals resources to
have "guaranteed" qos
- exposes the metrics port and the fileserver port
- supports two rbac modes for the controller - ClusterRole and Role
To install:
```
helm upgrade --install pulumi-kubernetes-operator ./deploy/helm/pulumi-operator
```
### Related issues (optional)
Closes #684
<!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes
#1234'.
Or link to full URLs to issues or pull requests in other GitHub
repositories. -->
### Proposed changes This PR removes the custom `min` and `max` functions for the `time.Duration` type. Go v1.21 introduced built-in functions so we can rely on them. This PR also adds a simple test case for the existing `exactlyOneOf` function.
<!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md)
for Pulumi's contribution guidelines.
Help us merge your changes more quickly by adding more details such
as labels, milestones, and reviewers.-->
### Proposed changes
<!--Give us a brief description of what you've done and what it solves.
-->
Implements good defaults for the workspace resource, using a
["burstable"](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#burstable)
approach.
Since a workspace pod's utilization is bursty - with low resource usage
during idle times and with high resource usage during deployment ops -
the pod requests a small amount of resources (64mb, 100m) to be able to
idle. A deployment op is able to use much more memory - all available
memory on the host.
Users may customize the resources (e.g. to apply different requests
and/or limits). For large/complex Pulumi apps, it might make sense to
reserve more memory and/or use
#694.
The agent takes some pains to stay within the requested amount, using a
programmatic form of the
[GOMEMLIMIT](https://weaviate.io/blog/gomemlimit-a-game-changer-for-high-memory-applications)
environment variable. The agent detects the requested amount via the
Downward API. We don't use `GOMEMLIMIT` to avoid propagating it to
sub-processes, and because the format is a Kubernetes 'quantity'.
It was observed that zombies weren't being cleaned up, and this was
leading to resource exhaustion. Fixed by using
[tini](https://github.com/krallin/tini/) as the entrypoint process (PID
1).
### Related issues (optional)
<!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes
#1234'.
Or link to full URLs to issues or pull requests in other GitHub
repositories. -->
Closes #698
### Proposed changes This PR exposes the metrics service in our manifests and adds new unit tests to ensure the metrics logic is correct. ### Technical changes 1. Added `sigs.k8s.io/controller-runtime/pkg/metrics/filters` dependency to allow metrics to be secure by default (guidance from kubebuilder book/scaffold) 2. Adds new kustomize patches and bases to expose the metrics server in a typical PKO deployment 3. Refactor the `updateStackCallback` function 4. Added the `stacks_reconciling` metric 5. Added ginkgo tests to exercise the Program and Stack metrics 6. Fixed some bugs in our metrics as determined through the unit tests ### Testing - Manually tested a deployed PKO installation with the Prometheus Operator - Added unit tests (using ginkgo) Note, e2e tests will be added in a followup PR. ### Related issues (optional) Closes: #576
Turns out we already plumb target from Stack → Update → Agent → Auto API. This adds a small E2E test around the behavior.
Currently, if our automation APIs call fail they return non-nil errors to the operator. In #676 I modified `Update` to translate these errors into a "failed" status on the Update/Stack, but other operations (preview etc.) still surface these errors and automatically re-queue. We'd like to retry these failed updates much less aggressively than we retry transient network errors, for example. To accomplish this we do a few things: * We consolidate the update controller's streaming logic for consistent error handling across all operations. * We return errors with known gRPC status codes as-is, but unknown status codes are translated into failed results for all operations. * We start tracking the number of times a stack has attempted an update. This is used to determine how much exponential backoff to apply. * A failed update is considered synced for a cooldown period before we retry it. The cooldown period starts at 5 minutes and doubles for every failed attempt, eventually maxing out at 24 hours. Fixes #677
…ion (#712) <!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md) for Pulumi's contribution guidelines. Help us merge your changes more quickly by adding more details such as labels, milestones, and reviewers.--> ### Overview This PR implements an authentication and authorization layer for the agent's RPC endpoint. Authentication is performed by authenticating a bearer token via the TokenReview API. The operator uses its built-in service account token. Authorization is performed via the SubjectAccessReview API, which checks for following RBAC permission: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role rules: - apiGroups: - auto.pulumi.com resources: - workspaces/rpc verbs: - use ``` The workspace pod's service account must be granted the `system:auth-delegator` role using a `ClusterRoleBinding`. For. convenience, the installer creates a service account named `pulumi` into the `default` namespace, with an associated binding. The operator itself is granted the necessary permission to access the RPC endpoint. ### Proposed changes <!--Give us a brief description of what you've done and what it solves. --> - [x] agent grpc interceptor - [x] agent command args (`--auth-mode=kube`, `--kube-workspace-name=random-yaml`) - [x] operator client credentials - [x] operator RBAC permissions - [ ] ~cluster role binding for workspace service account (to ClusterRole named `system:auth-delegator`)~ - [x] install a "default/pulumi" service account with RBAC - [x] Provide a Flux sample network policy - [x] Update the e2e test manifests to have requisite account, rbac, and network policy. ### Future Enhancement This implementation uses the operator's default service account token, but to further improve security it should use an audience-scoped token, where the audience is the agent service address as opposed to the API server. Such tokens may be created by the operator with a call to TokenRequest, and checked with TokenReview by adding the expected audience to the context (`authenticator.WithAudience`). ### Related issues (optional) <!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes #1234'. Or link to full URLs to issues or pull requests in other GitHub repositories. --> Closes #609 #### Examples Some example requests: ``` random-yaml-workspace-0 pulumi 2024-10-09T21:09:43.905Z INFO cmd.serve.grpc finished unary call with code OK {"grpc.start_time": "2024-10-09T21:09:43Z", "grpc.request.deadline": "2024-10-09T21:59:43Z", "system": "grpc", "span.kind": "server", "grpc.service": "agent.AutomationService", "grpc.method": "WhoAmI", "user.id": "81be050c-9ad4-4708-9a52-413064700747", "user.name": "system:serviceaccount:default:dev", "peer.address": "127.0.0.1:56394", "auth.mode": "kubernetes", "grpc.code": "OK", "grpc.time_ms": 441.086} random-yaml-workspace-0 pulumi 2024-10-09T21:09:52.934Z INFO cmd.serve.grpc finished unary call with code Unauthenticated {"grpc.start_time": "2024-10-09T21:09:52Z", "grpc.request.deadline": "2024-10-09T21:59:52Z", "system": "grpc", "span.kind": "server", "grpc.service": "agent.AutomationService", "grpc.method": "WhoAmI", "peer.address": "127.0.0.1:57380", "auth.mode": "kubernetes", "error": "rpc error: code = Unauthenticated desc = Request unauthenticated with Bearer", "grpc.code": "Unauthenticated", "grpc.time_ms": 0.095} ```
<!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md)
for Pulumi's contribution guidelines.
Help us merge your changes more quickly by adding more details such
as labels, milestones, and reviewers.-->
### Proposed changes
<!--Give us a brief description of what you've done and what it solves.
-->
This PR optimizes the watches to reduce the likelihood of a race between
the cache and the reconciliation loop.
The general idea is to use predicates to filter on the 'edge' events
that would allow the reconciler to make forward progress. For example,
the workspace reconciler applies a StatefulSet, then waits for it to be
ready. Only when the StatefulSet is indeed ready should reconciliation
be queued. Before this PR, the watch on StatefulSet was triggering upon
any revision change (way too eager).
Overall, a number of optimizations were implemented:
- workspace controller should wait for the statefulset to be ready
- workspace controller need not watch the Service
- stack controller should wait for flux source to be ready
- stack controller should not trigger reconciliation upon adding its
finalizer (which increments the generation).
- stack controller should wait for the update to complete
- update controller should wait for workspace readiness
- update controller should not enqueue the completed updates of a given
workspace
Also, some debug statements were introduced to reason about the revision
changes over time, since the "conflicts" are detected based on revision.
In particular, we emit a debug statement when reconciliation is
triggered by a given watch, to understand 'why'.
### Related issues (optional)
<!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes
#1234'.
Or link to full URLs to issues or pull requests in other GitHub
repositories. -->
Closes #700
Supercedes #679
### Example
Here's the result of applying
[operator/examples/random-yaml/stack.yaml](https://github.com/pulumi/pulumi-kubernetes-operator/blob/78836991b2691034b2c029df4040ea2016240cd8/operator/examples/random-yaml/stack.yaml).
#### Before
```log
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:26.032Z ERROR updating status {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "841b4d5b-029c-4457-a2b7-8c1821475adf", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile.func1
controller-manager-6f849dcb6b-4wscw manager /go/operator/internal/controller/auto/workspace_controller.go:110
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile
controller-manager-6f849dcb6b-4wscw manager /go/operator/internal/controller/auto/workspace_controller.go:181
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:26.033Z ERROR Reconciler error {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "841b4d5b-029c-4457-a2b7-8c1821475adf", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:26.047Z ERROR unable to save object status {"controller": "stack-controller", "namespace": "default", "name": "random-yaml", "reconcileID": "6a99e260-ca2d-4376-b95f-ab490acbe32a", "error": "Operation cannot be fulfilled on stacks.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/pulumi.(*StackReconciler).Reconcile.func1
controller-manager-6f849dcb6b-4wscw manager /go/operator/internal/controller/pulumi/stack_controller.go:445
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/pulumi.(*StackReconciler).Reconcile
controller-manager-6f849dcb6b-4wscw manager /go/operator/internal/controller/pulumi/stack_controller.go:741
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:26.047Z ERROR Reconciler error {"controller": "stack-controller", "namespace": "default", "name": "random-yaml", "reconcileID": "6a99e260-ca2d-4376-b95f-ab490acbe32a", "error": "Operation cannot be fulfilled on stacks.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:26.091Z ERROR updating status {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "d397d7d1-c74b-43fe-b4a0-80c3d39ee3ea", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile.func1
controller-manager-6f849dcb6b-4wscw manager /go/operator/internal/controller/auto/workspace_controller.go:110
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile
controller-manager-6f849dcb6b-4wscw manager /go/operator/internal/controller/auto/workspace_controller.go:188
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:26.091Z ERROR Reconciler error {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "d397d7d1-c74b-43fe-b4a0-80c3d39ee3ea", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:26.111Z ERROR updating status {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "b94710c8-8165-4937-a99c-a4e065f63788", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile.func1
controller-manager-6f849dcb6b-4wscw manager /go/operator/internal/controller/auto/workspace_controller.go:110
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile
controller-manager-6f849dcb6b-4wscw manager /go/operator/internal/controller/auto/workspace_controller.go:188
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:26.111Z ERROR Reconciler error {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "b94710c8-8165-4937-a99c-a4e065f63788", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:32.082Z ERROR updating status {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "7cb6660a-508c-4392-80cd-749fe72def9f", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile.func1
controller-manager-6f849dcb6b-4wscw manager /go/operator/internal/controller/auto/workspace_controller.go:110
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile
controller-manager-6f849dcb6b-4wscw manager /go/operator/internal/controller/auto/workspace_controller.go:252
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:32.082Z ERROR Reconciler error {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "7cb6660a-508c-4392-80cd-749fe72def9f", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:36.114Z ERROR updating status {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "480ad23a-2d02-40a6-b4a4-198fb9df01c4", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile.func1
controller-manager-6f849dcb6b-4wscw manager /go/operator/internal/controller/auto/workspace_controller.go:110
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile
controller-manager-6f849dcb6b-4wscw manager /go/operator/internal/controller/auto/workspace_controller.go:316
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:36.114Z ERROR Reconciler error {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "480ad23a-2d02-40a6-b4a4-198fb9df01c4", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
```
#### After
```log
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:45.212Z INFO Reconciling Stack {"controller": "stack-controller", "namespace": "default", "name": "random-yaml", "reconcileID": "585499fc-fb33-4505-8594-6a9fd8c810b2", "revision": "893372"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:45.225Z INFO KubeAPIWarningLogger metadata.finalizers: "finalizer.stack.pulumi.com": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:45.225Z INFO installing watcher for newly seen source kind {"GroupVersionKind": "source.toolkit.fluxcd.io/v1, Kind=GitRepository"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:45.225Z INFO Starting EventSource {"controller": "stack-controller", "source": "kind source: *unstructured.Unstructured"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:45.238Z INFO Reconciling Workspace {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "d1595cfb-0fe6-46d5-9e9c-2227f334eb7f", "revision": "893374"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:45.238Z INFO Applying StatefulSet {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "d1595cfb-0fe6-46d5-9e9c-2227f334eb7f", "revision": "893374", "hash": "608b7e170ccd95754972cde1178ab426", "source": {"Generation":1,"ForceRequest":"","Git":null,"Flux":{"Url":"http://source-controller.flux-system.svc.cluster.local./gitrepository/default/pulumi-examples/1432c2ce26299516223c0ad228e92294eba4b2c1.tar.gz","Digest":"sha256:44d554df090dcdeb9bae908cd38155c8c93db05f64dc72982027e8e1294af0d3","Dir":"random-yaml/"}}}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:48.609Z INFO Reconciling Workspace {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "5e7b9b87-fab5-4da0-b025-65dadefc097e", "revision": "893380"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:48.609Z INFO Applying StatefulSet {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "5e7b9b87-fab5-4da0-b025-65dadefc097e", "revision": "893380", "hash": "608b7e170ccd95754972cde1178ab426", "source": {"Generation":1,"ForceRequest":"","Git":null,"Flux":{"Url":"http://source-controller.flux-system.svc.cluster.local./gitrepository/default/pulumi-examples/1432c2ce26299516223c0ad228e92294eba4b2c1.tar.gz","Digest":"sha256:44d554df090dcdeb9bae908cd38155c8c93db05f64dc72982027e8e1294af0d3","Dir":"random-yaml/"}}}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:48.719Z INFO Connecting to workspace pod {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "5e7b9b87-fab5-4da0-b025-65dadefc097e", "revision": "893380", "addr": "random-yaml-workspace.default.svc.cluster.local:50051"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:48.724Z INFO Connected to workspace pod {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "5e7b9b87-fab5-4da0-b025-65dadefc097e", "revision": "893380", "addr": "random-yaml-workspace.default.svc.cluster.local:50051"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:48.724Z INFO Running pulumi install {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "5e7b9b87-fab5-4da0-b025-65dadefc097e", "revision": "893380"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:51.509Z INFO Creating Pulumi stack(s) {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "5e7b9b87-fab5-4da0-b025-65dadefc097e", "revision": "893413"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:55.250Z INFO workspace pod initialized {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "5e7b9b87-fab5-4da0-b025-65dadefc097e", "revision": "893422"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:55.250Z INFO Ready {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "5e7b9b87-fab5-4da0-b025-65dadefc097e", "revision": "893422"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:55.255Z INFO Reconciling Stack {"controller": "stack-controller", "namespace": "default", "name": "random-yaml", "reconcileID": "910a2faa-5b49-4590-a23a-d9ef8cab0023", "revision": "893375"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:55.271Z INFO Reconciling Update {"controller": "update", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"random-yaml-b7kng2l5","namespace":"default"}, "namespace": "default", "name": "random-yaml-b7kng2l5", "reconcileID": "5b9ced73-989f-46fa-a485-746686519832"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:55.271Z INFO Updating the status {"controller": "update", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"random-yaml-b7kng2l5","namespace":"default"}, "namespace": "default", "name": "random-yaml-b7kng2l5", "reconcileID": "5b9ced73-989f-46fa-a485-746686519832"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:55.275Z INFO Connecting {"controller": "update", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"random-yaml-b7kng2l5","namespace":"default"}, "namespace": "default", "name": "random-yaml-b7kng2l5", "reconcileID": "5b9ced73-989f-46fa-a485-746686519832", "addr": "random-yaml-workspace.default.svc.cluster.local:50051"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:55.278Z INFO Selecting the stack {"controller": "update", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"random-yaml-b7kng2l5","namespace":"default"}, "namespace": "default", "name": "random-yaml-b7kng2l5", "reconcileID": "5b9ced73-989f-46fa-a485-746686519832", "stackName": "dev"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:57.333Z INFO Applying the update {"controller": "update", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"random-yaml-b7kng2l5","namespace":"default"}, "namespace": "default", "name": "random-yaml-b7kng2l5", "reconcileID": "5b9ced73-989f-46fa-a485-746686519832", "type": "up"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:57.334Z INFO Executing update operation {"controller": "update", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"random-yaml-b7kng2l5","namespace":"default"}, "namespace": "default", "name": "random-yaml-b7kng2l5", "reconcileID": "5b9ced73-989f-46fa-a485-746686519832", "request": "message:\"Stack Update (up)\" target_dependents:false refresh:true"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:02:05.421Z INFO Update complete {"controller": "update", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"random-yaml-b7kng2l5","namespace":"default"}, "namespace": "default", "name": "random-yaml-b7kng2l5", "reconcileID": "5b9ced73-989f-46fa-a485-746686519832", "result": "stdout:\"Updating (dev)\\n\\nView Live: https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4536\\n\\n+ pulumi:pulumi:Stack: (create)\\n [urn=urn:pulumi:dev::random::pulumi:pulumi:Stack::random-dev]\\n + random:index/randomPassword:RandomPassword: (create)\\n [urn=urn:pulumi:dev::random::random:index/randomPassword:RandomPassword::randomPassword]\\n length : 16\\n overrideSpecial: \\\"_%@\\\"\\n special : true\\n --outputs:--\\n bcryptHash : [secret]\\n id : \\\"none\\\"\\n lower : true\\n minLower : 0\\n minNumeric : 0\\n minSpecial : 0\\n minUpper : 0\\n number : true\\n numeric : true\\n result : [secret]\\n upper : true\\n --outputs:--\\n password: [secret]\\nResources:\\n + 2 created\\n\\nDuration: 3s\\n\" summary:{start_time:{seconds:1729044119} end_time:{seconds:1729044122} result:\"succeeded\" message:\"\\\"Stack Update (up)\\\"\"} permalink:\"https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4536\" outputs:{key:\"password\" value:{value:\"\\\"fAMcmXI2DfW7KCey\\\"\" secret:true}}"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:02:05.438Z INFO Reconciling Stack {"controller": "stack-controller", "namespace": "default", "name": "random-yaml", "reconcileID": "a100c4f3-c5b9-4894-8644-fabc1f71d008", "revision": "893437"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:02:05.546Z INFO Commit hash unchanged. Will wait for Source update or resync. {"controller": "stack-controller", "namespace": "default", "name": "random-yaml", "reconcileID": "a100c4f3-c5b9-4894-8644-fabc1f71d008", "revision": "893437"}
```
This adds `applyconfiguration-gen` to generate apply configurations for our APIs. The workspace template is updated to use it. It's a moderately uphill battle to get apply configurations working with controllers due to various pieces not including built-in support for them. In practice this means we need to cut a couple small corners -- in particular implementing our own `DeepCopy` and tweaking the generated `GroupVersion` code. Fixes #704.
We currently issue one `SetAllConfig` RPC for each user-specified config. This is slow but it has important correctness guarantees: 1. The order we apply config matters -- if the user specifies `foo: foo` followed by `foo: bar`, the net result must always be `foo: bar`. 2. The Pulumi CLI (and therefore Automation API) only allows specifying `--path` on an all-or-nothing basis. This is bad for us because we potentially have a blend of path and non-path keys. Ideally we would be able to supply all of our configs to the automation API in a single call, and in the case where _all_ of our config keys are path-like (or all are not path-like) we actually can do that because we no longer have limitation (2). This PR makes that possible in the general case by transforming our config keys in a way that allows us to treat them as if they are all path-like. In particular: * The agent's `SetAllConfig` handler is modified to take a list of configs instead of a map in order to preserve config order. The top-level `path` param is also removed and handled on a per-key basis. * While resolving configs, we escape any non-path keys so subsequent path parsing treats them as verbatim. For example `foo.bar` gets escaped as `["foo.bar"]`. * We can then supply all of our keys at once to Automation API with `Path: true`. * If there are no configs to set then the operator doesn't invoke `SetAllConfig`. Fixes #650
<!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md)
for Pulumi's contribution guidelines.
Help us merge your changes more quickly by adding more details such
as labels, milestones, and reviewers.-->
### Proposed changes
<!--Give us a brief description of what you've done and what it solves.
-->
Use a sequential numbering strategy for the stack update objects.
Before:
```
NAME WORKSPACE PROGRESSING FAILED COMPLETE URL
random-yaml-tf9hxngl random-yaml
random-yaml-tf9hxngl random-yaml True False False
random-yaml-tf9hxngl random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4814
random-yaml-vqzmzxtr random-yaml
random-yaml-vqzmzxtr random-yaml True False False
random-yaml-vqzmzxtr random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4815
random-yaml-m4wm7dzt random-yaml
random-yaml-m4wm7dzt random-yaml True False False
random-yaml-m4wm7dzt random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4816
```
After:
```
NAME WORKSPACE PROGRESSING FAILED COMPLETE URL
random-yaml-1929cadf4a8 random-yaml
random-yaml-1929cadf4a8 random-yaml True False False
random-yaml-1929cadf4a8 random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4828
random-yaml-1929caf016f random-yaml
random-yaml-1929caf016f random-yaml True False False
random-yaml-1929caf016f random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4829
random-yaml-1929cb00a8f random-yaml
random-yaml-1929cb00a8f random-yaml True False False
random-yaml-1929cb00a8f random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4830
```
When you list the completed updates, they're now in order:
```
❯ kubectl get update
NAME WORKSPACE PROGRESSING FAILED COMPLETE URL
random-yaml-1929cadf4a8 random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4828
random-yaml-1929caf016f random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4829
random-yaml-1929cb00a8f random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4830
random-yaml-1929cb11670 random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4831
random-yaml-1929cb22165 random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4832
random-yaml-1929cb32d64 random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4833
random-yaml-1929cb43738 random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4834
random-yaml-1929cb540cb random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4835
```
### Related issues (optional)
<!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes
#1234'.
Or link to full URLs to issues or pull requests in other GitHub
repositories. -->
Closes #710
<!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md)
for Pulumi's contribution guidelines.
Help us merge your changes more quickly by adding more details such
as labels, milestones, and reviewers.-->
### Proposed changes
<!--Give us a brief description of what you've done and what it solves.
-->
Adds a top-level `serviceAccountName` to the Stack spec, since it is
practically a required element for every stack.
Note that the `workspaceTemplate` would take precedence.
The Workspace applies the default value of `default` if none is
specified.
Side-effect: some outdated manifests were regenerated.
### Related issues (optional)
<!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes
#1234'.
Or link to full URLs to issues or pull requests in other GitHub
repositories. -->
Closes #720
<!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md)
for Pulumi's contribution guidelines.
Help us merge your changes more quickly by adding more details such
as labels, milestones, and reviewers.-->
### Proposed changes
<!--Give us a brief description of what you've done and what it solves.
-->
Use 25 by default to facilitate the blocking operations in the auto API,
and support the `MAX_CONCURRENT_RECONCILES` environment variable.
### Related issues (optional)
<!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes
#1234'.
Or link to full URLs to issues or pull requests in other GitHub
repositories. -->
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #725 +/- ##
===========================================
+ Coverage 8.76% 50.01% +41.24%
===========================================
Files 4 30 +26
Lines 1164 4061 +2897
===========================================
+ Hits 102 2031 +1929
- Misses 1055 1846 +791
- Partials 7 184 +177 ☔ View full report in Codecov by Sentry. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Blog post: https://www.pulumi.com/blog/pulumi-kubernetes-operator-2-0/
See merge commit: #586