Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: Rancher pod killed during backup restore when pruning is enabled #532

Closed
mmartin24 opened this issue Jul 25, 2024 · 6 comments
Closed

BUG: Rancher pod killed during backup restore when pruning is enabled #532

mmartin24 opened this issue Jul 25, 2024 · 6 comments
Assignees
@mallardduck
Labels
kind/bug team/observability&backup

Comments

@mmartin24
Copy link

mmartin24 commented Jul 25, 2024 •
edited by olblak
Loading

Issue

While restoring a backup in Rancher with the exact same configuration as in the previous one, the rancher pod gets killed and Rancher UI dissappears as can be seen in the video

Screencast.from.25-07-24.16.40.15.webm

Step to reproduce (long way)

  • Deploy Rancher 2.8.5
  • Deploy a git repo
  • Create a Backup using the Backup app on Rancher (preferably on S3)
  • Create a new Rancher 2.8.5
  • Deploy default Backup app
  • Restore previously backed image WITH prune enabled (marked and recommended by default)

Short version to reproduce

  • Deploy Rancher 2.8.5
  • Create the following secret
kubectl create secret generic aws-secret \
  --from-literal=accessKey=xxx \
  --from-literal=secretKey=yyy
  • Deploy default Backup app with this info:
bucketName: epinio-ci
  credentialSecretName: aws-secret
  credentialSecretNamespace: default
  enabled: true
  folder: mmt-rancher-backup
  endpoint: s3.eu-central-1.amazonaws.com
  endpointCA:
  insecureTLSSkipVerify: false
  region: eu-central-1

URL: https://github.com/rancher/fleet-examples
Branch: master
Path: simple

Observed Behavior

Rancher pod gets deleted and although tries to recreate is never able to do so

Screencast.from.25-07-24.16.41.42.webm

Expected behavior

Pod should be able to recover well.
Gitjob is correctly deployed and active

Additional info

  • This affects Rancher 2.7-head, 2.8-head and 2.9-head, however, it seemed not to affect Rancher 2.7.6(at least when tried for anoher issue)
  • If the recovery is done within the same cluster that the back was taken, the UI survives
  • If the restore is done WITHOUT pruning, the UI survives, although ocassionally the gitjob is in a waiting status

Testing environment

  • Single cluster k3s with k8s version v1.27.10+k3s1
@mmartin24 mmartin24 mentioned this issue Jul 25, 2024
Closed
@manno
Copy link
Member

manno commented Jul 26, 2024 •
edited
Loading

This seems unrelated to fleet. Not sure if this goes into the rancher repo or https://github.com/rancher/backup-restore-operator/issues

While the restore seems to result in an inconsistent state, I'm wondering why this is fatal. I think it's related to this code path https://github.com/rancher/rancher/blob/release/v2.9/pkg/catalogv2/git/download.go#L78
We need better logs from the rancher pod, but maybe the "ignore error" case doesn't match the git CLI's message due to an update? stern -n cattle-system -l "app=rancher" captures logs across restarts.

@mmartin24
Copy link
Author

mmartin24 commented Jul 30, 2024 •
edited
Loading

This seems unrelated to fleet. Not sure if this goes into the rancher repo or https://github.com/rancher/backup-restore-operator/issues

While the restore seems to result in an inconsistent state, I'm wondering why this is fatal. I think it's related to this code path https://github.com/rancher/rancher/blob/release/v2.9/pkg/catalogv2/git/download.go#L78 We need better logs from the rancher pod, but maybe the "ignore error" case doesn't match the git CLI's message due to an update? stern -n cattle-system -l "app=rancher" captures logs across restarts.

Thanks for the tip.

Full Logs here 
→ stern -n cattle-system -l "app=rancher"        
+ rancher-7648944cfd-6pz5z › rancher
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:26 [INFO] Rancher version v2.9-6e9cf233a87796ee9a96e73c096a82101fc9c2ff-head (6e9cf233a87796ee9a96e73c096a82101fc9c2ff) is starting
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:26 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features: ClusterRegistry:}
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:26 [INFO] Listening on /tmp/log.sock
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:26 [INFO] EnsureSecretForServiceAccount: waiting for secret [rancher-token-l4bsq] to be populated with token
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:26 [INFO] Running in clustered mode with ID 10.42.0.58, monitoring endpoint cattle-system/rancher
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:26 [INFO] Applying CRD features.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:27 [INFO] Updating embedded CRD clusterroletemplatebindings.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:27 [INFO] Updating embedded CRD globalroles.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:27 [INFO] Updating embedded CRD globalrolebindings.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:27 [INFO] Updating embedded CRD projects.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:27 [INFO] Updating embedded CRD projectroletemplatebindings.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:27 [INFO] Updating embedded CRD roletemplates.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:27 [INFO] Updating embedded CRD clusterproxyconfigs.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:27 [INFO] Updating embedded CRD uiplugins.catalog.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:28 [INFO] Applying CRD navlinks.ui.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:28 [INFO] Applying CRD podsecurityadmissionconfigurationtemplates.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:28 [INFO] Applying CRD clusters.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:28 [INFO] Applying CRD apiservices.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:28 [INFO] Applying CRD clusterregistrationtokens.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:29 [INFO] Applying CRD settings.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:29 [INFO] Applying CRD preferences.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:29 [INFO] Applying CRD features.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:29 [INFO] Applying CRD clusterrepos.catalog.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:29 [INFO] Applying CRD operations.catalog.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:29 [INFO] Applying CRD apps.catalog.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:29 [INFO] Applying CRD fleetworkspaces.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:29 [INFO] Applying CRD managedcharts.management.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:30 [INFO] Applying CRD clusters.provisioning.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:30 [INFO] Applying CRD clusters.provisioning.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:30 [INFO] Applying CRD rkeclusters.rke.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:30 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:30 [INFO] Applying CRD rkebootstraps.rke.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:30 [INFO] Applying CRD rkebootstraptemplates.rke.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:30 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:31 [INFO] Applying CRD custommachines.rke.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:31 [INFO] Applying CRD etcdsnapshots.rke.cattle.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:31 [INFO] Applying CRD clusters.cluster.x-k8s.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:31 [INFO] Applying CRD machinedeployments.cluster.x-k8s.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:31 [INFO] Applying CRD machinehealthchecks.cluster.x-k8s.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:31 [INFO] Applying CRD machines.cluster.x-k8s.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:31 [INFO] Applying CRD machinesets.cluster.x-k8s.io
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=GlobalDns controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting /v1, Kind=Secret controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=RkeAddon controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=NodeTemplate controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplateVersion controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting /v1, Kind=ConfigMap controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=KontainerDriver controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=ClusterCatalog controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=ProjectCatalog controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=Catalog controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=RoleTemplate controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting API controllers
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplate controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting apiextensions.k8s.io/v1, Kind=CustomResourceDefinition controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=Project controller
rancher-7648944cfd-6pz5z rancher I0730 13:50:40.840784      39 leaderelection.go:250] attempting to acquire leader lease kube-system/cattle-controllers...
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterAppRevision controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=GlobalRole controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting rke.cattle.io/v1, Kind=RKEBootstrap controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=Token controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting catalog.cattle.io/v1, Kind=UIPlugin controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplate controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting /v1, Kind=Endpoints controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting cluster.x-k8s.io/v1beta1, Kind=Machine controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting provisioning.cattle.io/v1, Kind=Cluster controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=DynamicSchema controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting project.cattle.io/v3, Kind=App controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=ClusterRegistrationToken controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplateRevision controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=APIService controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting catalog.cattle.io/v1, Kind=ClusterRepo controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=Group controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=ProjectRoleTemplateBinding controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterApp controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=Setting controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=NodePool controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=ClusterProxyConfig controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=ClusterRoleTemplateBinding controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sSystemImage controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=NodeDriver controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=User controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting /v1, Kind=Namespace controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=Feature controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=AuthConfig controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=Preference controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sServiceOption controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=Node controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting management.cattle.io/v3, Kind=GlobalRoleBinding controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting /v1, Kind=ServiceAccount controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting cluster controllers for local
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting /v1, Kind=Namespace controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting /v1, Kind=Secret controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting /v1, Kind=ServiceAccount controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:40 [INFO] Starting cluster agent for local [owner=false]
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Starting /v1, Kind=ConfigMap controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Starting management.cattle.io/v3, Kind=User controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Starting management.cattle.io/v3, Kind=Group controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Starting management.cattle.io/v3, Kind=Token controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Starting /v1, Kind=Secret controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Listening on :443
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Listening on :80
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca@1721720268,O=dynamiclistener-org: notBefore=2024-07-23 07:37:48 +0000 UTC notAfter=2025-07-30 13:50:41 +0000 UTC
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [WARNING] dynamiclistener [::]:443: no cached certificate available for preload - deferring certificate load until storage initialization or first client request
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Active TLS secret / (ver=) (count 5): map[listener.cattle.io/cn-10.100.103.149.nip.io:10.100.103.149.nip.io listener.cattle.io/cn-10.42.0.58:10.42.0.58 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=B293C2862B69B3AF4CB068C747E1453135F1D708]
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Active TLS secret cattle-system/serving-cert (ver=21344) (count 6): map[field.cattle.io/projectId:local:p-l5xkq listener.cattle.io/cn-10.100.103.149.nip.io:10.100.103.149.nip.io listener.cattle.io/cn-10.42.0.13:10.42.0.13 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=82E93477C7A5CA427F0160DF2B42A0989A7F5FED]
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Listening on :444
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Active TLS secret cattle-system/tls-rancher-internal (ver=21358) (count 3): map[field.cattle.io/projectId:local:p-l5xkq listener.cattle.io/cn-10.42.0.13:10.42.0.13 listener.cattle.io/cn-10.43.14.46:10.43.14.46 listener.cattle.io/fingerprint:SHA1=66DC729E217186D1A9E166EFCDE0D812DF444FE1]
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca@1721720270,O=dynamiclistener-org: notBefore=2024-07-23 07:37:50 +0000 UTC notAfter=2025-07-30 13:50:41 +0000 UTC
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca@1721720270,O=dynamiclistener-org: notBefore=2024-07-23 07:37:50 +0000 UTC notAfter=2025-07-30 13:50:41 +0000 UTC
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Updating TLS secret for cattle-system/tls-rancher-internal (count: 4): map[field.cattle.io/projectId:local:p-l5xkq listener.cattle.io/cn-10.42.0.13:10.42.0.13 listener.cattle.io/cn-10.43.14.46:10.43.14.46 listener.cattle.io/cn-10.43.25.16:10.43.25.16 listener.cattle.io/fingerprint:SHA1=8BFA2458E15FCE43586870FCC5D665A447E510F9]
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Active TLS secret cattle-system/tls-rancher-internal (ver=21502) (count 4): map[field.cattle.io/projectId:local:p-l5xkq listener.cattle.io/cn-10.42.0.13:10.42.0.13 listener.cattle.io/cn-10.43.14.46:10.43.14.46 listener.cattle.io/cn-10.43.25.16:10.43.25.16 listener.cattle.io/fingerprint:SHA1=8BFA2458E15FCE43586870FCC5D665A447E510F9]
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca@1721720268,O=dynamiclistener-org: notBefore=2024-07-23 07:37:48 +0000 UTC notAfter=2025-07-30 13:50:41 +0000 UTC
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Updating TLS secret for cattle-system/serving-cert (count: 7): map[field.cattle.io/projectId:local:p-l5xkq listener.cattle.io/cn-10.100.103.149.nip.io:10.100.103.149.nip.io listener.cattle.io/cn-10.42.0.13:10.42.0.13 listener.cattle.io/cn-10.42.0.58:10.42.0.58 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=AD824DDEB9D5ABD58ABF0366AA9141E135939B8A]
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Starting /v1, Kind=Secret controller
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Updating TLS secret for cattle-system/tls-rancher-internal (count: 4): map[field.cattle.io/projectId:local:p-l5xkq listener.cattle.io/cn-10.42.0.13:10.42.0.13 listener.cattle.io/cn-10.43.14.46:10.43.14.46 listener.cattle.io/cn-10.43.25.16:10.43.25.16 listener.cattle.io/fingerprint:SHA1=8BFA2458E15FCE43586870FCC5D665A447E510F9]
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Active TLS secret cattle-system/serving-cert (ver=21503) (count 7): map[field.cattle.io/projectId:local:p-l5xkq listener.cattle.io/cn-10.100.103.149.nip.io:10.100.103.149.nip.io listener.cattle.io/cn-10.42.0.13:10.42.0.13 listener.cattle.io/cn-10.42.0.58:10.42.0.58 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=AD824DDEB9D5ABD58ABF0366AA9141E135939B8A]
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [INFO] Updating TLS secret for cattle-system/serving-cert (count: 7): map[field.cattle.io/projectId:local:p-l5xkq listener.cattle.io/cn-10.100.103.149.nip.io:10.100.103.149.nip.io listener.cattle.io/cn-10.42.0.13:10.42.0.13 listener.cattle.io/cn-10.42.0.58:10.42.0.58 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=AD824DDEB9D5ABD58ABF0366AA9141E135939B8A]
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:41 [ERROR] error syncing 'rancher-rke2-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-rke2-charts/675f1b63a0a83905972dcab2794479ed599a6f41b86cd6193d69472d0fa889c9 fetch origin -- 9d90836f0d7ce1075b8eeb3a208c2bbb88466086 error: exit status 128, detail: error: Server does not allow request for unadvertised object 9d90836f0d7ce1075b8eeb3a208c2bbb88466086
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher W0730 13:50:42.083092      39 warnings.go:70] v1 ComponentStatus is deprecated in v1.19+
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- f1595b195e7bedfeb758e8ea6385d91c5b7a5549 error: exit status 128, detail: error: Server does not allow request for unadvertised object f1595b195e7bedfeb758e8ea6385d91c5b7a5549
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=ServersTransport
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=Cluster
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for cluster.x-k8s.io/v1beta1, Kind=MachineSet
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKEBootstrapTemplate
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectRoleTemplateBinding
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateVersion
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=Amazonec2MachineTemplate
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=GroupMember
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.io/v1alpha1, Kind=TLSOption
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for resources.cattle.io/v1, Kind=Restore
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=User
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=RancherUserNotification
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=IngressRouteUDP
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=IngressRouteTCP
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSINode
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for /v1, Kind=Pod
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for /v1, Kind=PodTemplate
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRoleBinding
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine-config.cattle.io/v1, Kind=AzureConfig
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.io/v1alpha1, Kind=TraefikService
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=AzureMachine
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for networking.k8s.io/v1, Kind=Ingress
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=IngressRoute
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=ClusterRepo
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityAdmissionConfigurationTemplate
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistrationToken
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterAppRevision
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDnsProvider
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for apiregistration.k8s.io/v1, Kind=APIService
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for /v1, Kind=ConfigMap
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKEControlPlane
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=TraefikService
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for cert-manager.io/v1, Kind=Certificate
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for /v1, Kind=ReplicationController
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=VmwarevsphereMachine
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for storage.k8s.io/v1, Kind=StorageClass
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for discovery.k8s.io/v1, Kind=EndpointSlice
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for k3s.cattle.io/v1, Kind=Addon
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.io/v1alpha1, Kind=IngressRoute
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine-config.cattle.io/v1, Kind=VmwarevsphereConfig
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for resources.cattle.io/v1, Kind=ResourceSet
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine-config.cattle.io/v1, Kind=Amazonec2Config
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeDriver
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for policy/v1, Kind=PodDisruptionBudget
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=Preference
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Bundle
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for project.cattle.io/v3, Kind=AppRevision
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for cert-manager.io/v1, Kind=CertificateRequest
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for batch/v1, Kind=CronJob
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for apps/v1, Kind=Deployment
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=Group
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRoleTemplateBinding
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=App
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for cluster.x-k8s.io/v1beta1, Kind=Machine
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for /v1, Kind=PersistentVolumeClaim
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRole
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for apiextensions.k8s.io/v1, Kind=CustomResourceDefinition
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=DynamicSchema
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ImageScan
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.io/v1alpha1, Kind=IngressRouteTCP
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for ui.cattle.io/v1, Kind=NavLink
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterApp
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.io/v1alpha1, Kind=ServersTransport
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for /v1, Kind=Endpoints
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for autoscaling/v2, Kind=HorizontalPodAutoscaler
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=HarvesterMachine
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sSystemImage
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=Amazonec2Machine
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for /v1, Kind=Event
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.io/v1alpha1, Kind=ServersTransportTCP
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for networking.k8s.io/v1, Kind=IngressClass
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for project.cattle.io/v3, Kind=App
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for coordination.k8s.io/v1, Kind=Lease
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=Template
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplate
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSIStorageCapacity
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=SamlToken
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for /v1, Kind=Node
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for scheduling.k8s.io/v1, Kind=PriorityClass
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRole
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=LinodeMachine
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectCatalog
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for /v1, Kind=Namespace
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for k3s.cattle.io/v1, Kind=ETCDSnapshotFile
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRegistrationToken
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for /v1, Kind=LimitRange
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKEBootstrap
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=FleetWorkspace
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for resources.cattle.io/v1, Kind=Backup
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for networking.k8s.io/v1, Kind=NetworkPolicy
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKECluster
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for events.k8s.io/v1, Kind=Event
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine-config.cattle.io/v1, Kind=HarvesterConfig
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine-config.cattle.io/v1, Kind=DigitaloceanConfig
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplateVersion
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeTemplate
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=AzureMachineTemplate
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sServiceOption
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine-config.cattle.io/v1, Kind=LinodeConfig
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=APIService
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for helm.cattle.io/v1, Kind=HelmChart
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=Token
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodePool
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for /v1, Kind=Service
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=RoleBinding
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for apps/v1, Kind=DaemonSet
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDns
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for cert-manager.io/v1, Kind=Issuer
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleDeployment
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for flowcontrol.apiserver.k8s.io/v1beta3, Kind=PriorityLevelConfiguration
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=Node
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplateRevision
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.io/v1alpha1, Kind=MiddlewareTCP
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=LinodeMachineTemplate
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for apps/v1, Kind=ControllerRevision
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke.cattle.io/v1, Kind=CustomMachine
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=TLSStore
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterProxyConfig
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=Project
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for helm.cattle.io/v1, Kind=HelmChartConfig
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=KontainerDriver
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepo
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for cluster.x-k8s.io/v1beta1, Kind=MachineHealthCheck
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=Role
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for provisioning.cattle.io/v1, Kind=Cluster
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=Operation
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplate
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for /v1, Kind=ResourceQuota
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistration
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=RoleTemplate
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for certificates.k8s.io/v1, Kind=CertificateSigningRequest
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for /v1, Kind=PersistentVolume
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for apps/v1, Kind=StatefulSet
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=DigitaloceanMachine
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=UIPlugin
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=Setting
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Order
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for storage.k8s.io/v1, Kind=VolumeAttachment
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Content
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for flowcontrol.apiserver.k8s.io/v1beta3, Kind=FlowSchema
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterCatalog
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=VmwarevsphereMachineTemplate
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepoRestriction
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for cert-manager.io/v1, Kind=ClusterIssuer
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for apps/v1, Kind=ReplicaSet
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Cluster
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for cluster.x-k8s.io/v1beta1, Kind=Cluster
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateContent
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSIDriver
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=EtcdBackup
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for /v1, Kind=Secret
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.io/v1alpha1, Kind=IngressRouteUDP
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.io/v1alpha1, Kind=TLSStore
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=UserAttribute
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=Catalog
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectNetworkPolicy
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleNamespaceMapping
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke.cattle.io/v1, Kind=ETCDSnapshot
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=ComposeConfig
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=Middleware
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for /v1, Kind=ServiceAccount
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for batch/v1, Kind=Job
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for cluster.x-k8s.io/v1beta1, Kind=MachineDeployment
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.io/v1alpha1, Kind=Middleware
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for node.k8s.io/v1, Kind=RuntimeClass
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeAddon
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=Feature
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=TLSOption
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=MiddlewareTCP
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=HarvesterMachineTemplate
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterGroup
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=AuthConfig
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for management.cattle.io/v3, Kind=ManagedChart
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Challenge
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [ERROR] error syncing 'rancher-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721 fetch origin -- 3230fedb5609dfa7f17b726c2b778de381275d9c error: exit status 128, detail: error: Server does not allow request for unadvertised object 3230fedb5609dfa7f17b726c2b778de381275d9c
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:42 [ERROR] error syncing 'rancher-rke2-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-rke2-charts/675f1b63a0a83905972dcab2794479ed599a6f41b86cd6193d69472d0fa889c9 fetch origin -- 9d90836f0d7ce1075b8eeb3a208c2bbb88466086 error: exit status 128, detail: error: Server does not allow request for unadvertised object 9d90836f0d7ce1075b8eeb3a208c2bbb88466086
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:43 [ERROR] error syncing 'rancher-rke2-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-rke2-charts/675f1b63a0a83905972dcab2794479ed599a6f41b86cd6193d69472d0fa889c9 fetch origin -- 9d90836f0d7ce1075b8eeb3a208c2bbb88466086 error: exit status 128, detail: fatal: unable to access 'https://git.rancher.io/rke2-charts/': The requested URL returned error: 500
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:43 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- f1595b195e7bedfeb758e8ea6385d91c5b7a5549 error: exit status 128, detail: error: Server does not allow request for unadvertised object f1595b195e7bedfeb758e8ea6385d91c5b7a5549
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:44 [ERROR] error syncing 'rancher-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721 fetch origin -- 3230fedb5609dfa7f17b726c2b778de381275d9c error: exit status 128, detail: error: Server does not allow request for unadvertised object 3230fedb5609dfa7f17b726c2b778de381275d9c
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:45 [ERROR] error syncing 'rancher-rke2-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-rke2-charts/675f1b63a0a83905972dcab2794479ed599a6f41b86cd6193d69472d0fa889c9 fetch origin -- 9d90836f0d7ce1075b8eeb3a208c2bbb88466086 error: exit status 128, detail: error: Server does not allow request for unadvertised object 9d90836f0d7ce1075b8eeb3a208c2bbb88466086
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:46 [ERROR] error syncing 'rancher-rke2-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-rke2-charts/675f1b63a0a83905972dcab2794479ed599a6f41b86cd6193d69472d0fa889c9 fetch origin -- 9d90836f0d7ce1075b8eeb3a208c2bbb88466086 error: exit status 128, detail: error: Server does not allow request for unadvertised object 9d90836f0d7ce1075b8eeb3a208c2bbb88466086
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:46 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- f1595b195e7bedfeb758e8ea6385d91c5b7a5549 error: exit status 128, detail: error: Server does not allow request for unadvertised object f1595b195e7bedfeb758e8ea6385d91c5b7a5549
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:46 [ERROR] error syncing 'rancher-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721 fetch origin -- 3230fedb5609dfa7f17b726c2b778de381275d9c error: exit status 128, detail: error: Server does not allow request for unadvertised object 3230fedb5609dfa7f17b726c2b778de381275d9c
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:46 [ERROR] error syncing 'rancher-rke2-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-rke2-charts/675f1b63a0a83905972dcab2794479ed599a6f41b86cd6193d69472d0fa889c9 fetch origin -- 9d90836f0d7ce1075b8eeb3a208c2bbb88466086 error: exit status 128, detail: error: Server does not allow request for unadvertised object 9d90836f0d7ce1075b8eeb3a208c2bbb88466086
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:47 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- f1595b195e7bedfeb758e8ea6385d91c5b7a5549 error: exit status 128, detail: error: Server does not allow request for unadvertised object f1595b195e7bedfeb758e8ea6385d91c5b7a5549
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:47 [ERROR] error syncing 'rancher-rke2-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-rke2-charts/675f1b63a0a83905972dcab2794479ed599a6f41b86cd6193d69472d0fa889c9 fetch origin -- 9d90836f0d7ce1075b8eeb3a208c2bbb88466086 error: exit status 128, detail: error: Server does not allow request for unadvertised object 9d90836f0d7ce1075b8eeb3a208c2bbb88466086
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:48 [ERROR] error syncing 'rancher-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721 fetch origin -- 3230fedb5609dfa7f17b726c2b778de381275d9c error: exit status 128, detail: error: Server does not allow request for unadvertised object 3230fedb5609dfa7f17b726c2b778de381275d9c
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:48 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- f1595b195e7bedfeb758e8ea6385d91c5b7a5549 error: exit status 128, detail: error: Server does not allow request for unadvertised object f1595b195e7bedfeb758e8ea6385d91c5b7a5549
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:49 [ERROR] error syncing 'rancher-rke2-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-rke2-charts/675f1b63a0a83905972dcab2794479ed599a6f41b86cd6193d69472d0fa889c9 fetch origin -- 9d90836f0d7ce1075b8eeb3a208c2bbb88466086 error: exit status 128, detail: error: Server does not allow request for unadvertised object 9d90836f0d7ce1075b8eeb3a208c2bbb88466086
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:49 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- f1595b195e7bedfeb758e8ea6385d91c5b7a5549 error: exit status 128, detail: error: Server does not allow request for unadvertised object f1595b195e7bedfeb758e8ea6385d91c5b7a5549
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:49 [ERROR] error syncing 'rancher-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721 fetch origin -- 3230fedb5609dfa7f17b726c2b778de381275d9c error: exit status 128, detail: error: Server does not allow request for unadvertised object 3230fedb5609dfa7f17b726c2b778de381275d9c
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:50 [ERROR] error syncing 'rancher-rke2-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-rke2-charts/675f1b63a0a83905972dcab2794479ed599a6f41b86cd6193d69472d0fa889c9 fetch origin -- 9d90836f0d7ce1075b8eeb3a208c2bbb88466086 error: exit status 128, detail: error: Server does not allow request for unadvertised object 9d90836f0d7ce1075b8eeb3a208c2bbb88466086
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:51 [ERROR] error syncing 'rancher-rke2-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-rke2-charts/675f1b63a0a83905972dcab2794479ed599a6f41b86cd6193d69472d0fa889c9 fetch origin -- 9d90836f0d7ce1075b8eeb3a208c2bbb88466086 error: exit status 128, detail: error: Server does not allow request for unadvertised object 9d90836f0d7ce1075b8eeb3a208c2bbb88466086
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:51 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- f1595b195e7bedfeb758e8ea6385d91c5b7a5549 error: exit status 128, detail: error: Server does not allow request for unadvertised object f1595b195e7bedfeb758e8ea6385d91c5b7a5549
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:51 [ERROR] error syncing 'rancher-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721 fetch origin -- 3230fedb5609dfa7f17b726c2b778de381275d9c error: exit status 128, detail: error: Server does not allow request for unadvertised object 3230fedb5609dfa7f17b726c2b778de381275d9c
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:52 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- f1595b195e7bedfeb758e8ea6385d91c5b7a5549 error: exit status 128, detail: error: Server does not allow request for unadvertised object f1595b195e7bedfeb758e8ea6385d91c5b7a5549
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:52 [ERROR] error syncing 'rancher-rke2-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-rke2-charts/675f1b63a0a83905972dcab2794479ed599a6f41b86cd6193d69472d0fa889c9 fetch origin -- 9d90836f0d7ce1075b8eeb3a208c2bbb88466086 error: exit status 128, detail: error: Server does not allow request for unadvertised object 9d90836f0d7ce1075b8eeb3a208c2bbb88466086
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:53 [ERROR] error syncing 'rancher-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721 fetch origin -- 3230fedb5609dfa7f17b726c2b778de381275d9c error: exit status 128, detail: error: Server does not allow request for unadvertised object 3230fedb5609dfa7f17b726c2b778de381275d9c
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:53 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- f1595b195e7bedfeb758e8ea6385d91c5b7a5549 error: exit status 128, detail: error: Server does not allow request for unadvertised object f1595b195e7bedfeb758e8ea6385d91c5b7a5549
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:54 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- f1595b195e7bedfeb758e8ea6385d91c5b7a5549 error: exit status 128, detail: error: Server does not allow request for unadvertised object f1595b195e7bedfeb758e8ea6385d91c5b7a5549
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:55 [ERROR] error syncing 'rancher-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721 fetch origin -- 3230fedb5609dfa7f17b726c2b778de381275d9c error: exit status 128, detail: error: Server does not allow request for unadvertised object 3230fedb5609dfa7f17b726c2b778de381275d9c
rancher-7648944cfd-6pz5z rancher , requeuing
rancher-7648944cfd-6pz5z rancher 2024/07/30 13:50:55 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: ensure failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- f1595b195e7bedfeb758e8ea6385d91c5b7a5549 error: exit status 128, detail: error: Server does not allow request for unadvertised object f1595b195e7bedfeb758e8ea6385d91c5b7a5549
rancher-7648944cfd-6pz5z rancher , requeuing
- rancher-7648944cfd-6pz5z › rancher

@mmartin24
Copy link
Author

@kkaempf , as per Mario's comment above could you please advice where should this issue be transferred? (Rancher or Backup Restore Operator)
Thank you!

@kkaempf kkaempf transferred this issue from rancher/fleet Jul 30, 2024
@kkaempf
Copy link

kkaempf commented Jul 30, 2024

ping @MKlimuszka - asking for help

@mallardduck
Copy link
Member

The Rancher pod going down during a prune is expected behaviour for an in-place restore. The prune setting is true by default to facilitate in-place restores and should be false for migrations. From reading the short steps and the longer steps it's a bit unclear if this process should be a migration or an in-place restore.

This part of the short steps makes me think a migration:

Create a Backup using the Backup app on Rancher (preferably on S3)
Create a new Rancher 2.8.5

As does this:

If the recovery is done within the same cluster that the back was taken, the UI survives

Overall my advice would be to follow the migration documentation to successfully restore the backup on the new cluster. This restore is done without rancher installed/running at all, and once successfully restored you start rancher. After which point you should create new backups for the rancher app running on this cluster.

From a "In-place restore" vs "Migration" point of view, the primary question to ask is "what cluster did Rancher live on when this backup was taken?" If the answer is "a different one", then it's a migration; otherwise it can be an in-place restore. Even though the backup was successfully restored once upon migration, the backup was still created on a different cluster so should not be used for in-place restores.

@mmartin24
Copy link
Author

The Rancher pod going down during a prune is expected behaviour for an in-place restore. The prune setting is true by default to facilitate in-place restores and should be false for migrations. From reading the short steps and the longer steps it's a bit unclear if this process should be a migration or an in-place restore.

This part of the short steps makes me think a migration:

Create a Backup using the Backup app on Rancher (preferably on S3)
Create a new Rancher 2.8.5

As does this:

If the recovery is done within the same cluster that the back was taken, the UI survives

Overall my advice would be to follow the migration documentation to successfully restore the backup on the new cluster. This restore is done without rancher installed/running at all, and once successfully restored you start rancher. After which point you should create new backups for the rancher app running on this cluster.

From a "In-place restore" vs "Migration" point of view, the primary question to ask is "what cluster did Rancher live on when this backup was taken?" If the answer is "a different one", then it's a migration; otherwise it can be an in-place restore. Even though the backup was successfully restored once upon migration, the backup was still created on a different cluster so should not be used for in-place restores.

Thanks @mallardduck .
The issue was raised under the wrong assumption that migration could be done the same way of restoration on the same cluster. Not being the case, I understand this issue does not apply anymore. Closing it.

@github-project-automation github-project-automation bot moved this from 🆕 New to ✅ Done in Fleet Jul 31, 2024
@mallardduck mallardduck self-assigned this Jul 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mallardduck mallardduck

Labels
kind/bug team/observability&backup
Projects
Fleet
Archived in project
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kkaempf @manno @mallardduck @mmartin24