409 conflict error

[10000-ki]

ENV

• k8s version : 1.23
• java operator sdk version : 4.4.0

ERROR Message

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://ai1.io.navercorp.com:6443/apis/apps/v1/namespaces/opensearchoperatorsecuritye2e-enablesecuritytest-4976a0a5-b09a/deployments/os-test-opensearch-cluster-dashboards?fieldManager=e2etestopensearchreconciler&force=true. Message: Operation cannot be fulfilled on deployments.apps "os-test-opensearch-cluster-dashboards": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=apps, kind=deployments, name=os-test-opensearch-cluster-dashboards, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on deployments.apps "os-test-opensearch-cluster-dashboards": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).

there was an issue that the resource version continued to be upgraded even though there was no spec modification in the deployment
Because of this, infinite reconcile loop execution occurred, which resulted in 409 errors..

[csviri]

@10000-ki could you tell more context? Are you using dependent resources, maybe would be great to have a small reproducer if it hard to describe when it exactly happens.

Note that resource version changes even on status and metadata change not just spec.

[csviri]

I see this is related to deployments, see also this: #2249
But by default should not cause issue. You can verify if you problem is related by setting this feature flag to false:

java-operator-sdk/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/api/config/ConfigurationService.java

Line 375 in 4e42612

default boolean previousAnnotationForDependentResourcesEventFiltering() {

[csviri]

Sorry, now I see that you are using v4.4.0 where this was not present yet. (see https://javaoperatorsdk.io/docs/v4-5-migration )

[10000-ki]

could you tell more context? Are you using dependent resources, maybe would be great to have a small reproducer if it hard to describe when it exactly happens.

yes i use dependent resources

@ControllerConfiguration(
    labelSelector = "app.kubernetes.io/managed-by=$OPERATOR_LABEL_V1",
    dependents = [
        Dependent(type = OpenSearchCertificatesSecret::class, name = "OpenSearchCertificatesSecret", useEventSourceWithName = SECRET_EVENT_SOURCE),
        Dependent(type = OpenSearchConfigMap::class, name = "OpenSearchConfigMap", useEventSourceWithName = CONFIGMAP_EVENT_SOURCE),
        Dependent(type = OpenSearchSecurityConfigMap::class, name = "OpenSearchSecurityConfigMap", useEventSourceWithName = CONFIGMAP_EVENT_SOURCE),
        Dependent(type = OpenSearchService::class, name = "OpenSearchService", useEventSourceWithName = SERVICE_EVENT_SOURCE),
        Dependent(type = OpenSearchDiscoveryService::class, name = "OpenSearchDiscoveryService", useEventSourceWithName = SERVICE_EVENT_SOURCE),
        Dependent(type = OpenSearchNodeStatefulSet::class, name = "OpenSearchNodeStatefulSet", useEventSourceWithName = STS_EVENT_SOURCE),
        Dependent(type = OpenSearchNodeDeployment::class, name = "OpenSearchNodeDeployment", useEventSourceWithName = DEPLOYMENT_EVENT_SOURCE),
        // @formatter:off
        Dependent(type = OpenSearchDashboardsConfigMap::class, name = "OpenSearchDashboardsConfigMap", useEventSourceWithName = CONFIGMAP_EVENT_SOURCE), // ktlint-disable
        Dependent(type = OpenSearchDashboardsService::class, name = "OpenSearchDashboardsService", useEventSourceWithName = SERVICE_EVENT_SOURCE), // ktlint-disable
        Dependent(type = OpenSearchDashboardsDeployment::class, name = "OpenSearchDashboardsDeployment", useEventSourceWithName = DEPLOYMENT_EVENT_SOURCE), // ktlint-disable
        Dependent(type = OpenSearchClusterManagerPodDisruptionBudget::class, name = "OpenSearchClusterManagerPodDisruptionBudget", useEventSourceWithName = PDB_EVENT_SOURCE), // ktlint-disable
        // @formatter:on
    ],
)

A change occurs only in OpenSearchCR Spec And this has nothing to do with OpenSearchDashboardsDeployment
but the reconcile event will continue to run about OpenSearchDashboardsDeployment

[10000-ki]

is it related to this issue?
kubernetes/kubernetes#95460

[csviri]

Yes it seems to be that is an issue on that k8s version you are using from this issue. Pls try also to update to newer version of JOSDK that might also solve this (if you use dependent resources)

[10000-ki]

okay i will upgrade lib version

[10000-ki]

@csviri hi can i ask something??

i saw this pr
https://github.com/operator-framework/java-operator-sdk/pull/2008/files

If the resource version is different, it is impossible to update, so I think the SSA method is adjusting the resource version between actual and target

May I know why you deleted it here?

[csviri]

The reason is that there should not be a need for optimistic locking for SSA, since that already handles the ownership of the fields. It is assumed that the desired won't have a resource version set, so there is no optimistic locking happening in this case.

[shawkins]

I think it was also a consequence of the annotation based tracking of updates as well as it doesn't need to be locked for correctness.

[csviri]

I think it was also a consequence of the annotation based tracking of updates as well as it doesn't need to be locked for correctness.

At the end it was not needed with event recorder either I think (just was much easier to reason about, also it was before a non-ssa based update, so that is more reasonable there too)

[10000-ki]

if I can't upgrade k8s version to the next 1.23 version (internal problem), are there any other ways??

Should I use the CSA instead of the SSA?

[csviri]

@10000-ki yes, please try that

[10000-ki]

thank you for guide I'll close this issue