Report error when any registry problem happen

[nf3lix]

Description

I'm facing an issue when trying to use camel-k on an Azure Kubernetes Cluster with GitLab as container registry.
When I create an integration from a file, the created resource stays in phase "Building Kit" for a while until it fails (phase "Error").
There's no pod created for the integration. I was able to run the same integration on a minikube cluster with internal registry.

There are multiple potential reasons for that behavior, including a misconfiguration of the container registry.
However, troubleshooting is difficult because there aren't many logs created by either the operator or the integration itself.
I couldn't find a way to extend the logging.

Steps to reproduce

It's not possible to mention all steps in detail because of the specific setup. The most important steps I went through are the following:

1. Create a namespace "sample-namespace"
2. Use kamel cli to install camel-k:
   kamel install --namespace sample-namespace --registry <internal-registry> --organization <internal-organization> --registry-secret <registry-token> --build-publish-strategy=Spectrum --force
3. Create the integration mentioned below
4. Apply the file using kubectl apply -f sample-route.yaml

apiVersion: camel.apache.org/v1
kind: Integration
metadata:
  name: sample-route
spec:
  flows:
    - route:
        id: route-a1ba
        from:
          uri: kamelet:timer-source
          id: from-8fbf
          parameters:
            message: hello world
            period: 5000
          steps:
            - log:
                message: ${body}
                id: log-d8d4

This is the only reasonably meaningful error message I could finde:
"errorVerbose":"unable to find integration container: sample-route\nerror during trait customization"

The full logs generated by the integration and the operator are listed below.
Does anyone have any idea either why the build fails or how to extend the logging?
Of course I don't expect the underlying problem to be solved by this issue.
Rather, I am looking for a way to get more meaningful logs.

Logs from the integration:

  Type     Reason                       Age                  From                                Message
  ----     ------                       ----                 ----                                -------
  Normal   IntegrationConditionChanged  29m                  camel-k-integration-controller      Condition "IntegrationPlatformAvailable" is "True" for Integration sample-route: sample-namespace/camel-k
  Normal   ReasonRelatedObjectChanged   29m                  camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Running"
  Normal   ReasonRelatedObjectChanged   29m                  camel-k-integration-kit-controller  Integration Kit "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Build Running"
  Normal   ReasonRelatedObjectChanged   29m                  camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Pending"
  Normal   IntegrationConditionChanged  29m                  camel-k-integration-controller      Condition "IntegrationKitAvailable" is "False" for Integration sample-route: creating a new integration kit
  Normal   ReasonRelatedObjectChanged   29m                  camel-k-integration-kit-controller  Integration Kit "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Build Submitted"
  Normal   ReasonRelatedObjectChanged   29m                  camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Scheduling"
  Normal   ReasonRelatedObjectChanged   24m                  camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Failed"
  Normal   ReasonRelatedObjectChanged   24m                  camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Initialization" (recovery 1 of 5)
  Normal   ReasonRelatedObjectChanged   24m                  camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Scheduling" (recovery 1 of 5)
  Normal   ReasonRelatedObjectChanged   24m                  camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Pending" (recovery 1 of 5)
  Normal   ReasonRelatedObjectChanged   24m                  camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Running" (recovery 1 of 5)
  Normal   IntegrationConditionChanged  24m (x3 over 29m)    camel-k-integration-controller      Condition "KameletsAvailable" is "True" for Integration sample-route: kamelets [timer-source] found in repositories: (Kubernetes[namespace=sample-namespace], Empty[])
  Normal   IntegrationPhaseUpdated      24m (x3 over 29m)    camel-k-integration-controller      Integration "sample-route" in phase "Initialization"
  Normal   IntegrationPhaseUpdated      24m (x3 over 29m)    camel-k-integration-controller      Integration "sample-route" in phase "Building Kit"
  Normal   IntegrationConditionChanged  24m (x2 over 24m)    camel-k-integration-controller      Condition "IntegrationKitAvailable" is "False" for Integration sample-route: integration kit sample-namespace/kit-ch3tosbhiudc73db4760 is in state "Build Running"
  Normal   ReasonRelatedObjectChanged   19m                  camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Failed" (recovery 1 of 5)
  Normal   ReasonRelatedObjectChanged   14m (x9 over 19m)    camel-k-build-controller            (combined from similar events): Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Running" (recovery 3 of 5)
  Normal   ReasonRelatedObjectChanged   14m (x9 over 19m)    camel-k-build-controller            (combined from similar events): Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Running" (recovery 3 of 5)
  Normal   ReasonRelatedObjectChanged   12m                  camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Failed" (recovery 3 of 5)
  Normal   ReasonRelatedObjectChanged   12m                  camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Initialization" (recovery 4 of 5)
  Normal   ReasonRelatedObjectChanged   12m                  camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Scheduling" (recovery 4 of 5)
  Normal   ReasonRelatedObjectChanged   12m                  camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Pending" (recovery 4 of 5)
  Normal   ReasonRelatedObjectChanged   12m                  camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Running" (recovery 4 of 5)
  Normal   ReasonRelatedObjectChanged   7m1s                 camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Failed" (recovery 4 of 5)
  Normal   ReasonRelatedObjectChanged   7m1s                 camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Initialization" (recovery 5 of 5)
  Normal   ReasonRelatedObjectChanged   7m1s                 camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Scheduling" (recovery 5 of 5)
  Normal   ReasonRelatedObjectChanged   7m1s                 camel-k-build-controller            Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Pending" (recovery 5 of 5)
  Normal   ReasonRelatedObjectChanged   2m1s (x3 over 7m1s)  camel-k-build-controller            (combined from similar events): Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Error" (recovery 5 of 5)
  Normal   ReasonRelatedObjectChanged   2m1s (x3 over 7m1s)  camel-k-build-controller            (combined from similar events): Build "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Error" (recovery 5 of 5)
  Normal   ReasonRelatedObjectChanged   2m1s                 camel-k-integration-kit-controller  Integration Kit "kit-ch3tosbhiudc73db4760", created by Integration "sample-route", changed phase to "Error"
  Normal   IntegrationConditionChanged  2m1s                 camel-k-integration-controller      Condition "IntegrationKitAvailable" is "False" for Integration sample-route: integration kit sample-namespace/kit-ch3tosbhiudc73db4760 is in state "Error"
  Normal   IntegrationPhaseUpdated      2m1s                 camel-k-integration-controller      Integration "sample-route" in phase "Error"
  Warning  IntegrationError             39s (x15 over 2m1s)  camel-k-integration-controller      Cannot reconcile Integration sample-route: error during trait customization: unable to find integration container: sample-route

Logs from the operator:

{"level":"info","ts":1682432126.6430852,"logger":"camel-k.controller.integration","msg":"Reconciling Integration","request-namespace":"sample-namespace","request-name":"sample-route"}
{"level":"info","ts":1682432126.643163,"logger":"camel-k.controller.integration","msg":"Invoking action build-kit","request-namespace":"sample-namespace","request-name":"sample-route","api-version":"camel.apache.org/v1","kind":"Integration","ns":"sample-namespace","name":"sample-route"}
{"level":"info","ts":1682432127.5997233,"logger":"camel-k.controller.build","msg":"Build queue duration 599.68444ms","request-namespace":"sample-namespace","request-name":"sample-route","build-queue-duration":0.59968444,"api-version":"camel.apache.org/v1","kind":"Build","ns":"sample-namespace","name":"kit-ch3tosbhiudc73db4760"}
{"level":"info","ts":1682432427.0041478,"logger":"camel-k.controller.build","msg":"Build duration 5m0.004124345s","request-namespace":"sample-namespace","request-name":"sample-route","build-attempt":4,"build-result":"Failed","build-duration":300.004124345,"api-version":"camel.apache.org/v1","kind":"Build","ns":"sample-namespace","name":"kit-ch3tosbhiudc73db4760"}
{"level":"info","ts":1682432427.4130895,"logger":"camel-k.controller.build","msg":"Build queue duration 413.078156ms","request-namespace":"sample-namespace","request-name":"sample-route","build-queue-duration":0.413078156,"api-version":"camel.apache.org/v1","kind":"Build","ns":"sample-namespace","name":"kit-ch3tosbhiudc73db4760"}
{"level":"info","ts":1682432727.0053852,"logger":"camel-k.controller.build","msg":"Build duration 5m0.005363162s","request-namespace":"sample-namespace","request-name":"sample-route","build-attempt":5,"build-result":"Error","build-duration":300.005363162,"api-version":"camel.apache.org/v1","kind":"Build","ns":"sample-namespace","name":"kit-ch3tosbhiudc73db4760"}
{"level":"info","ts":1682432727.396065,"logger":"camel-k","msg":"Kit kit-ch3tosbhiudc73db4760 ready, notify integration: sample-route"}
{"level":"info","ts":1682432727.3961773,"logger":"camel-k.controller.integration","msg":"Reconciling Integration","request-namespace":"sample-namespace","request-name":"sample-route"}
{"level":"info","ts":1682432727.3962264,"logger":"camel-k.controller.integration","msg":"Invoking action build-kit","request-namespace":"sample-namespace","request-name":"sample-route","api-version":"camel.apache.org/v1","kind":"Integration","ns":"sample-namespace","name":"sample-route"}
{"level":"info","ts":1682432727.5326107,"logger":"camel-k.controller.integration","msg":"state transition","request-namespace":"sample-namespace","request-name":"sample-route","api-version":"camel.apache.org/v1","kind":"Integration","ns":"sample-namespace","name":"sample-route","phase-from":"Building Kit","phase-to":"Error"}
{"level":"info","ts":1682432727.532716,"logger":"camel-k.controller.integration","msg":"Reconciling Integration","request-namespace":"sample-namespace","request-name":"sample-route"}
{"level":"info","ts":1682432727.5327551,"logger":"camel-k.controller.integration","msg":"Invoking action monitor","request-namespace":"sample-namespace","request-name":"sample-route","api-version":"camel.apache.org/v1","kind":"Integration","ns":"sample-namespace","name":"sample-route"}
{"level":"error","ts":1682432727.6471384,"msg":"Reconciler error","controller":"integration-controller","controllerGroup":"camel.apache.org","controllerKind":"Integration","Integration":{"name":"sample-route","namespace":"sample-namespace"},"namespace":"sample-namespace","name":"sample-route","reconcileID":"51598263-9d2d-4390-965a-37830743585d","error":"error during trait customization: unable to find integration container: sample-route","errorVerbose":"unable to find integration container: sample-route\nerror during trait customization\ngithub.com/apache/camel-k/pkg/trait.Apply\n\tgithub.com/apache/camel-k/pkg/trait/trait.go:59\ngithub.com/apache/camel-k/pkg/controller/integration.(*monitorAction).Handle\n\tgithub.com/apache/camel-k/pkg/controller/integration/monitor.go:114\ngithub.com/apache/camel-k/pkg/controller/integration.(*reconcileIntegration).Reconcile\n\tgithub.com/apache/camel-k/pkg/controller/integration/integration_controller.go:361\ngithub.com/apache/camel-k/pkg/util/monitoring.(*instrumentedReconciler).Reconcile\n\tgithub.com/apache/camel-k/pkg/util/monitoring/controller.go:70\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234\nruntime.goexit\n\truntime/asm_amd64.s:1571","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234"}
{"level":"info","ts":1682432727.6524758,"logger":"camel-k.controller.integration","msg":"Reconciling Integration","request-namespace":"sample-namespace","request-name":"sample-route"}

[squakez]

Thanks for reporting. Yes, I think we need to expose some error details (at least some clear log mentioning the registry is failing). I'm editing the title of this issue to better capture the requirement and label as a feature for future development. If you're willing to contribute to the issue, feel free to ask any development advice.

[nf3lix]

It turned out there was an unintentionally applied networkpolicy blocking all requests by default, including those for docker hub and the external container registry. Thus, the operator was not able to build the integration.

[github-actions]

This issue has been automatically marked as stale due to 90 days of inactivity.
It will be closed if no further activity occurs within 15 days.
If you think that’s incorrect or the issue should never stale, please simply write any comment.
Thanks for your contributions!