Clean up "object has already been modified" warnings #700 #717

EronWright · 2024-10-15T23:03:23Z

Proposed changes

This PR optimizes the watches to reduce the likelihood of a race between the cache and the reconciliation loop.

The general idea is to use predicates to filter on the 'edge' events that would allow the reconciler to make forward progress. For example, the workspace reconciler applies a StatefulSet, then waits for it to be ready. Only when the StatefulSet is indeed ready should reconciliation be queued. Before this PR, the watch on StatefulSet was triggering upon any revision change (way too eager).

Overall, a number of optimizations were implemented:

workspace controller should wait for the statefulset to be ready
workspace controller need not watch the Service
stack controller should wait for flux source to be ready
stack controller should not trigger reconciliation upon adding its finalizer (which increments the generation).
stack controller should wait for the update to complete
update controller should wait for workspace readiness
update controller should not enqueue the completed updates of a given workspace

Also, some debug statements were introduced to reason about the revision changes over time, since the "conflicts" are detected based on revision. In particular, we emit a debug statement when reconciliation is triggered by a given watch, to understand 'why'.

Related issues (optional)

Closes #700
Supercedes #679

Example

Here's the result of applying operator/examples/random-yaml/stack.yaml.

Before

controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:26.032Z    ERROR   updating status {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "841b4d5b-029c-4457-a2b7-8c1821475adf", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile.func1
controller-manager-6f849dcb6b-4wscw manager     /go/operator/internal/controller/auto/workspace_controller.go:110
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile
controller-manager-6f849dcb6b-4wscw manager     /go/operator/internal/controller/auto/workspace_controller.go:181
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:26.033Z    ERROR   Reconciler error        {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "841b4d5b-029c-4457-a2b7-8c1821475adf", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:26.047Z    ERROR   unable to save object status    {"controller": "stack-controller", "namespace": "default", "name": "random-yaml", "reconcileID": "6a99e260-ca2d-4376-b95f-ab490acbe32a", "error": "Operation cannot be fulfilled on stacks.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/pulumi.(*StackReconciler).Reconcile.func1
controller-manager-6f849dcb6b-4wscw manager     /go/operator/internal/controller/pulumi/stack_controller.go:445
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/pulumi.(*StackReconciler).Reconcile
controller-manager-6f849dcb6b-4wscw manager     /go/operator/internal/controller/pulumi/stack_controller.go:741
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:26.047Z    ERROR   Reconciler error        {"controller": "stack-controller", "namespace": "default", "name": "random-yaml", "reconcileID": "6a99e260-ca2d-4376-b95f-ab490acbe32a", "error": "Operation cannot be fulfilled on stacks.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:26.091Z    ERROR   updating status {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "d397d7d1-c74b-43fe-b4a0-80c3d39ee3ea", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile.func1
controller-manager-6f849dcb6b-4wscw manager     /go/operator/internal/controller/auto/workspace_controller.go:110
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile
controller-manager-6f849dcb6b-4wscw manager     /go/operator/internal/controller/auto/workspace_controller.go:188
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:26.091Z    ERROR   Reconciler error        {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "d397d7d1-c74b-43fe-b4a0-80c3d39ee3ea", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:26.111Z    ERROR   updating status {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "b94710c8-8165-4937-a99c-a4e065f63788", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile.func1
controller-manager-6f849dcb6b-4wscw manager     /go/operator/internal/controller/auto/workspace_controller.go:110
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile
controller-manager-6f849dcb6b-4wscw manager     /go/operator/internal/controller/auto/workspace_controller.go:188
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:26.111Z    ERROR   Reconciler error        {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "b94710c8-8165-4937-a99c-a4e065f63788", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:32.082Z    ERROR   updating status {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "7cb6660a-508c-4392-80cd-749fe72def9f", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile.func1
controller-manager-6f849dcb6b-4wscw manager     /go/operator/internal/controller/auto/workspace_controller.go:110
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile
controller-manager-6f849dcb6b-4wscw manager     /go/operator/internal/controller/auto/workspace_controller.go:252
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:32.082Z    ERROR   Reconciler error        {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "7cb6660a-508c-4392-80cd-749fe72def9f", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:36.114Z    ERROR   updating status {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "480ad23a-2d02-40a6-b4a4-198fb9df01c4", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile.func1
controller-manager-6f849dcb6b-4wscw manager     /go/operator/internal/controller/auto/workspace_controller.go:110
controller-manager-6f849dcb6b-4wscw manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile
controller-manager-6f849dcb6b-4wscw manager     /go/operator/internal/controller/auto/workspace_controller.go:316
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
controller-manager-6f849dcb6b-4wscw manager 2024-10-16T01:47:36.114Z    ERROR   Reconciler error        {"controller": "workspace", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "480ad23a-2d02-40a6-b4a4-198fb9df01c4", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"random-yaml\": the object has been modified; please apply your changes to the latest version and try again"}
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
controller-manager-6f849dcb6b-4wscw manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
controller-manager-6f849dcb6b-4wscw manager     /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222

After


controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:45.212Z    INFO    Reconciling Stack       {"controller": "stack-controller", "namespace": "default", "name": "random-yaml", "reconcileID": "585499fc-fb33-4505-8594-6a9fd8c810b2", "revision": "893372"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:45.225Z    INFO    KubeAPIWarningLogger    metadata.finalizers: "finalizer.stack.pulumi.com": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:45.225Z    INFO    installing watcher for newly seen source kind   {"GroupVersionKind": "source.toolkit.fluxcd.io/v1, Kind=GitRepository"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:45.225Z    INFO    Starting EventSource    {"controller": "stack-controller", "source": "kind source: *unstructured.Unstructured"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:45.238Z    INFO    Reconciling Workspace   {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "d1595cfb-0fe6-46d5-9e9c-2227f334eb7f", "revision": "893374"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:45.238Z    INFO    Applying StatefulSet    {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "d1595cfb-0fe6-46d5-9e9c-2227f334eb7f", "revision": "893374", "hash": "608b7e170ccd95754972cde1178ab426", "source": {"Generation":1,"ForceRequest":"","Git":null,"Flux":{"Url":"http://source-controller.flux-system.svc.cluster.local./gitrepository/default/pulumi-examples/1432c2ce26299516223c0ad228e92294eba4b2c1.tar.gz","Digest":"sha256:44d554df090dcdeb9bae908cd38155c8c93db05f64dc72982027e8e1294af0d3","Dir":"random-yaml/"}}}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:48.609Z    INFO    Reconciling Workspace   {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "5e7b9b87-fab5-4da0-b025-65dadefc097e", "revision": "893380"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:48.609Z    INFO    Applying StatefulSet    {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "5e7b9b87-fab5-4da0-b025-65dadefc097e", "revision": "893380", "hash": "608b7e170ccd95754972cde1178ab426", "source": {"Generation":1,"ForceRequest":"","Git":null,"Flux":{"Url":"http://source-controller.flux-system.svc.cluster.local./gitrepository/default/pulumi-examples/1432c2ce26299516223c0ad228e92294eba4b2c1.tar.gz","Digest":"sha256:44d554df090dcdeb9bae908cd38155c8c93db05f64dc72982027e8e1294af0d3","Dir":"random-yaml/"}}}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:48.719Z    INFO    Connecting to workspace pod     {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "5e7b9b87-fab5-4da0-b025-65dadefc097e", "revision": "893380", "addr": "random-yaml-workspace.default.svc.cluster.local:50051"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:48.724Z    INFO    Connected to workspace pod      {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "5e7b9b87-fab5-4da0-b025-65dadefc097e", "revision": "893380", "addr": "random-yaml-workspace.default.svc.cluster.local:50051"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:48.724Z    INFO    Running pulumi install  {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "5e7b9b87-fab5-4da0-b025-65dadefc097e", "revision": "893380"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:51.509Z    INFO    Creating Pulumi stack(s)        {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "5e7b9b87-fab5-4da0-b025-65dadefc097e", "revision": "893413"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:55.250Z    INFO    workspace pod initialized       {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "5e7b9b87-fab5-4da0-b025-65dadefc097e", "revision": "893422"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:55.250Z    INFO    Ready   {"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"random-yaml","namespace":"default"}, "namespace": "default", "name": "random-yaml", "reconcileID": "5e7b9b87-fab5-4da0-b025-65dadefc097e", "revision": "893422"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:55.255Z    INFO    Reconciling Stack       {"controller": "stack-controller", "namespace": "default", "name": "random-yaml", "reconcileID": "910a2faa-5b49-4590-a23a-d9ef8cab0023", "revision": "893375"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:55.271Z    INFO    Reconciling Update      {"controller": "update", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"random-yaml-b7kng2l5","namespace":"default"}, "namespace": "default", "name": "random-yaml-b7kng2l5", "reconcileID": "5b9ced73-989f-46fa-a485-746686519832"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:55.271Z    INFO    Updating the status     {"controller": "update", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"random-yaml-b7kng2l5","namespace":"default"}, "namespace": "default", "name": "random-yaml-b7kng2l5", "reconcileID": "5b9ced73-989f-46fa-a485-746686519832"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:55.275Z    INFO    Connecting      {"controller": "update", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"random-yaml-b7kng2l5","namespace":"default"}, "namespace": "default", "name": "random-yaml-b7kng2l5", "reconcileID": "5b9ced73-989f-46fa-a485-746686519832", "addr": "random-yaml-workspace.default.svc.cluster.local:50051"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:55.278Z    INFO    Selecting the stack     {"controller": "update", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"random-yaml-b7kng2l5","namespace":"default"}, "namespace": "default", "name": "random-yaml-b7kng2l5", "reconcileID": "5b9ced73-989f-46fa-a485-746686519832", "stackName": "dev"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:57.333Z    INFO    Applying the update     {"controller": "update", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"random-yaml-b7kng2l5","namespace":"default"}, "namespace": "default", "name": "random-yaml-b7kng2l5", "reconcileID": "5b9ced73-989f-46fa-a485-746686519832", "type": "up"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:01:57.334Z    INFO    Executing update operation      {"controller": "update", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"random-yaml-b7kng2l5","namespace":"default"}, "namespace": "default", "name": "random-yaml-b7kng2l5", "reconcileID": "5b9ced73-989f-46fa-a485-746686519832", "request": "message:\"Stack Update (up)\"  target_dependents:false  refresh:true"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:02:05.421Z    INFO    Update complete {"controller": "update", "controllerGroup": "auto.pulumi.com", "controllerKind": "Update", "Update": {"name":"random-yaml-b7kng2l5","namespace":"default"}, "namespace": "default", "name": "random-yaml-b7kng2l5", "reconcileID": "5b9ced73-989f-46fa-a485-746686519832", "result": "stdout:\"Updating (dev)\\n\\nView Live: https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4536\\n\\n+ pulumi:pulumi:Stack: (create)\\n    [urn=urn:pulumi:dev::random::pulumi:pulumi:Stack::random-dev]\\n    + random:index/randomPassword:RandomPassword: (create)\\n        [urn=urn:pulumi:dev::random::random:index/randomPassword:RandomPassword::randomPassword]\\n        length         : 16\\n        overrideSpecial: \\\"_%@\\\"\\n        special        : true\\n        --outputs:--\\n        bcryptHash     : [secret]\\n        id             : \\\"none\\\"\\n        lower          : true\\n        minLower       : 0\\n        minNumeric     : 0\\n        minSpecial     : 0\\n        minUpper       : 0\\n        number         : true\\n        numeric        : true\\n        result         : [secret]\\n        upper          : true\\n    --outputs:--\\n    password: [secret]\\nResources:\\n    + 2 created\\n\\nDuration: 3s\\n\"  summary:{start_time:{seconds:1729044119}  end_time:{seconds:1729044122}  result:\"succeeded\"  message:\"\\\"Stack Update (up)\\\"\"}  permalink:\"https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/4536\"  outputs:{key:\"password\"  value:{value:\"\\\"fAMcmXI2DfW7KCey\\\"\"  secret:true}}"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:02:05.438Z    INFO    Reconciling Stack       {"controller": "stack-controller", "namespace": "default", "name": "random-yaml", "reconcileID": "a100c4f3-c5b9-4894-8644-fabc1f71d008", "revision": "893437"}
controller-manager-778bdbdfbb-cdrpb manager 2024-10-16T02:02:05.546Z    INFO    Commit hash unchanged. Will wait for Source update or resync.   {"controller": "stack-controller", "namespace": "default", "name": "random-yaml", "reconcileID": "a100c4f3-c5b9-4894-8644-fabc1f71d008", "revision": "893437"}

codecov · 2024-10-15T23:46:21Z

Codecov Report

Attention: Patch coverage is 16.66667% with 135 lines in your changes missing coverage. Please review.

Project coverage is 51.20%. Comparing base (7883699) to head (0c4e8cb).
Report is 1 commits behind head on v2.

Files with missing lines	Patch %	Lines
...tor/internal/controller/pulumi/stack_controller.go	23.33%	46 Missing ⚠️
...ator/internal/controller/auto/update_controller.go	2.38%	41 Missing ⚠️
...r/internal/controller/auto/workspace_controller.go	16.66%	19 Missing and 1 partial ⚠️
operator/internal/controller/pulumi/flux.go	33.33%	13 Missing and 3 partials ⚠️
operator/internal/controller/auto/utils.go	0.00%	12 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               v2     #717      +/-   ##
==========================================
- Coverage   53.00%   51.20%   -1.80%     
==========================================
  Files          29       29              
  Lines        3081     3199     +118     
==========================================
+ Hits         1633     1638       +5     
- Misses       1267     1379     +112     
- Partials      181      182       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

operator/internal/controller/auto/update_controller.go

operator/internal/controller/auto/workspace_controller.go

operator/internal/controller/pulumi/stack_controller.go

operator/internal/controller/pulumi/flux.go

operator/internal/controller/pulumi/utils.go

EronWright added 2 commits October 15, 2024 14:47

tighten up the watches in stack controller, workspace controller

160c16d

optimize flux source watch

6156231

EronWright requested review from blampe and rquitales October 15, 2024 23:03

EronWright changed the title ~~Issue 700~~ Clean up "object has already been modified" warnings #700 Oct 15, 2024

EronWright added the impact/no-changelog-required This issue doesn't require a CHANGELOG update label Oct 15, 2024

bugfix

18fa063

EronWright added 2 commits October 15, 2024 20:16

optimize update controller

78d0aaa

bugfix

c1bb90a

blampe approved these changes Oct 16, 2024

View reviewed changes

feedback

0c4e8cb

EronWright merged commit 2884fad into v2 Oct 17, 2024
7 checks passed

EronWright deleted the issue-700 branch October 17, 2024 02:02

mjeffryes assigned EronWright Oct 25, 2024

EronWright mentioned this pull request Oct 30, 2024

Extra reconciliation loops can cause harmless errors due to stale objects #30

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clean up "object has already been modified" warnings #700 #717

Clean up "object has already been modified" warnings #700 #717

EronWright commented Oct 15, 2024 •

edited

Loading

codecov bot commented Oct 15, 2024 •

edited

Loading

Clean up "object has already been modified" warnings #700 #717

Clean up "object has already been modified" warnings #700 #717

Conversation

EronWright commented Oct 15, 2024 • edited Loading

Proposed changes

Related issues (optional)

Example

Before

After

codecov bot commented Oct 15, 2024 • edited Loading

Codecov Report

EronWright commented Oct 15, 2024 •

edited

Loading

codecov bot commented Oct 15, 2024 •

edited

Loading