Skip to content

bug: cache does not get invalidated on resource deletion? #446

Open
@nazarewk

Description

@nazarewk

TLDR: I believe that with before-hook-creation under heavy load our ArgoCD (2.3.3 on EKS) processes a PreSync phase as Completed before fully processing resource (Job) deletion.

This is a followup from argoproj/argo-cd#10077

Probably watch event arrives late or gets stuck in a queue for a while and I could not track down any cache invalidation during deletion in argo-cd or gitops-engine itself.

// delete anything that need deleting
hooksPendingDeletion := createTasks.Filter(func(t *syncTask) bool { return t.deleteBeforeCreation() })
if hooksPendingDeletion.Len() > 0 {
ss := newStateSync(state)
for _, task := range hooksPendingDeletion {
t := task
ss.Go(func(state runState) runState {
sc.log.WithValues("dryRun", dryRun, "task", t).V(1).Info("Deleting")
if !dryRun {
err := sc.deleteResource(t)
if err != nil {
// it is possible to get a race condition here, such that the resource does not exist when
// delete is requested, we treat this as a nop
if !apierr.IsNotFound(err) {
state = failed
sc.setResourceResult(t, "", common.OperationError, fmt.Sprintf("failed to delete resource: %v", err))
}
} else {
// if there is anything that needs deleting, we are at best now in pending and
// want to return and wait for sync to be invoked again
state = pending
}
}
return state
})
}
state = ss.Wait()
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions