OKD 4.9 unable to delete projects

[bdlink]

On a cluster (vSphere UPI) that started with 4.6 and was upgraded to 4.9.0-0.okd-2022-02-12-140851 deletion of projects hangs in the terminating state. The knowledge base item at https://access.redhat.com/solutions/4165791 gives some suggestions which did not resolve the problem (which was likely introduced in the upgrade to 4.8 as the problem did not occur in 4.7 but has persisted in upgrades in 4.8 and 4.9).

The hanging seems to be caused by the kubernetes finalizer, indicating there is some resource that has not been deleted. Running the following command on one of the hung projects:
oc api-resources --verbs=list --namespaced -o name | xargs -n 1 oc get --show-kind --ignore-not-found -n $PROJECT_NAME

results in:
Error from server (InternalError): Internal error occurred: error resolving resource

and after a long delay:
NAME CATALOG AGE
packagemanifest.packages.operators.coreos.com/openshift-qiskit-operator Community Operators 486d
packagemanifest.packages.operators.coreos.com/kubernetes-imagepuller-operator Community Operators 486d
packagemanifest.packages.operators.coreos.com/volume-expander-operator Community Operators 486d
....
packagemanifest.packages.operators.coreos.com/eventing-kogito Community Operators 486d
packagemanifest.packages.operators.coreos.com/argocd-operator Community Operators 486d

The knowledge base article suggests deleting the resources output by the above command, but it is not clear how to do that in this case.

BTW, running must-gather creates a project which (after must-gather is complete) hangs in terminating state.

Sample Project status from a must-gather project:
status:
conditions:

• lastTransitionTime: "2022-05-04T18:12:37Z"
  message: All resources successfully discovered
  reason: ResourcesDiscovered
  status: "False"
  type: NamespaceDeletionDiscoveryFailure
• lastTransitionTime: "2021-12-21T20:04:40Z"
  message: All legacy kube types successfully parsed
  reason: ParsedGroupVersions
  status: "False"
  type: NamespaceDeletionGroupVersionParsingFailure
• lastTransitionTime: "2021-12-21T20:04:40Z"
  message: 'Failed to delete all resource types, 1 remaining: Internal error occurred:
  error resolving resource'
  reason: ContentDeletionFailed
  status: "True"
  type: NamespaceDeletionContentFailure
• lastTransitionTime: "2021-12-21T20:04:40Z"
  message: All content successfully removed
  reason: ContentRemoved
  status: "False"
  type: NamespaceContentRemaining
• lastTransitionTime: "2021-12-21T20:04:40Z"
  message: All content-preserving finalizers finished
  reason: ContentHasNoFinalizers
  status: "False"
  type: NamespaceFinalizersRemaining
  phase: Terminating

Suggestions how to find out what the undetected resources are?

[bdlink]

The api-resources that caused an internal error is runners.apps.gitlab.com which was coming from an operator that was present in OKD 4.7 but dissapeared by 4.9. Evidently the upgrade did not clean up the api-resource when upgrading at some point. Is not getting an internal error a bug?

oc get runners.apps.gitlab.com
Error from server (InternalError): Internal error occurred: error resolving resource

When I uninstalled the Strimzi operator, I notice there are still 10 strimzi-related api-resources present. Should uninstalling an operator also remove the related CRDs?

[bdlink]

Deleting the CRD for runners.apps.gitlab.com from the console failed with following message:

could not list instances: unable to find a custom resource client for runners.apps.gitlab.com: unable to load root certificates: unable to parse bytes as PEM block

[bdlink]

The Runner CRD had a bad field caBundle (see below). When this was fixed, the CRD was successfully deleted and the projects stuck in terminating were deleted.

The "bugs" would seem to be that

• there was no error attached to the CRD before I attempted to delete it. Making an edit (unrelated to the issue) and saving it caused no error
• oc get returns an internal error instead of a helpful error (like what is the name of the object it has issues with)
• perhaps an upgrade that removes an operator no longer supported should remove associated CRDs
• perhaps uninstalling an operator should clean up the associated CRDs

Unfortunately, it is not possible to know where the strange cert came from.

    strategy: Webhook
    webhook:
      clientConfig:
        service:
          namespace: gitlab-runner-system
          name: gitlab-runner-webhook-service
          path: /convert
          port: 443
        caBundle: Cg==
      conversionReviewVersions:
        - v1beta1

[bdlink]

BTW, this fix also fixes another problem of the collect-profile, image-pruner, and now (4.9?) ip-reconciler cronJobs spawning an infinite number of pods (over time) resulting in pod failure due to running out of resources.

Those three cron jobs created pods which did not get deleted after completion (or failure).

I am guessing the failure to delete these and stay within the cronJob limits was somehow related to the internal system error referenced at the top of the discussion. For example, collect-profile has the following retention spec:

  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1

with similar ones for the other two. In this case, the failure to automatically delete the pods can be successfully done manually with

oc delete pod --field-selector=status.phase==Succeeded --all-namespaces
oc delete pod --field-selector=status.phase==Failed --all-namespaces

I can't chase down the reason since this is no longer happening to me,