[v2.8] BRO not backing up namespaces created for FleetWorkspaces

[Daemonslayer2048]

Rancher Server Setup

• Rancher version: 2.8.1
• Installation option (Docker install/Helm Chart): Helm Install
• Kubernetes Version and Engine: v1.25.4+rke2r1

SURE-8919

Describe the bug
When using FleetWorkspaces in Rancher this will create a new namespace for said workspace. Due to this if a user attempts to restore on a new cluster the restore process will fail as it will not create said namespace.

To Reproduce
Steps to reproduce the behavior:

1. Create a fresh cluster
2. Create FleetWorkspaces (See additional context below)
3. Install backup operator and take a backup
4. Delete cluster
5. Restore Rancher on totally new cluster
6. Observe restore failure

Expected behavior
I would expect one of two things to happen:

1. Rancher restore should create the namespace as needed to allow the FleetWorkspace to be repopulated
2. Rancher restore will skip creating FleetWorkspaces as to not prevent the restore from completing
   Option one is preferable but two will at leas prevent end users from getting stuck.

Screenshots
Not needed

Additional context
Sample Fleet config

---
apiVersion: management.cattle.io/v3
kind: FleetWorkspace
metadata:
  name: enterprise
 
---
apiVersion: management.cattle.io/v3
kind: FleetWorkspace
metadata:
  name: edge

---
apiVersion: fleet.cattle.io/v1alpha1
kind: ClusterGroup
metadata:
  name: infra
  namespace: enterprise
spec:
  selector:
    matchExpressions: []
    matchLabels:
      infra: "true"
---
apiVersion: fleet.cattle.io/v1alpha1
kind: ClusterGroup
metadata:
  name: apps
  namespace: edge
spec:
  selector:
    matchExpressions: []
    matchLabels:
      apps: "true"

[ericpromislow]

Is this a dup of the other recent failure to save fleet secrets?

[Daemonslayer2048]

Do you happen to know what issue number that would be?

[ericpromislow]

rancher/rancher#44033

[dsmithbauer]

@ericpromislow I do not believe so. This ticket specifically addresses the problem where restoring a Rancher backup from the Backup Operator fails if additional Fleet workspaces are used (besides fleet-default) because the Fleet workspace namespaces are not recreated/stored in the backup. However, in the ticket you mention, we may want to also consider backing up secrets in any Fleet namespace (not just fleet-default). Otherwise, using the Rancher Backup Operator to restore a Rancher MCM cluster backup where Fleet is utilized to deploy services to downstream clusters does not work well and will fail without manual intervention (will not create Fleet workspaces and secrets).

[mallardduck]

@Daemonslayer2048 - Can you clarify a few things for us:

• When you tried this out, did the FleetWorkspace you created get backed up (is it in the backup file)?
• If yes, did an error occur during the restore because of that resource that caused it to fail restoring other resources?
• If yes, what other resources specifically failed to restore due to the FleetWorkspace failing to restore?

[jbiers]

@Daemonslayer2048 - During the triage process for this issue we were not able to reproduce the error. The FleetWorkspaces are being backed up and once restored into a new cluster, seemed to successfully recreate its namespace and the resources in it.

If you can, share with us the error message you get when trying to restore the backup and we can investigate it further. Also, make sure you have the default ResourceSet unedited in your local cluster. The questions asked by @mallardduck are also important to help us understand your case.

[jbiers]

Also, be sure you are following the migration instructions step by step as detailed in the docs here.

[Daemonslayer2048]

I believe some things may be getting mixed up. @mallardduck when you attempted to reproduce did you do a an etcd backup and restore, or just attempt a restore via the Rancher Backup? If you use an etcd backup and restore the namespace does get created so that is not the issue here.

@jbiers I did follow those docs exactly, the issue does not show up unless the ClusterGroups are namespaced.

[mallardduck]

@Daemonslayer2048 - No, i'm not referring to etcd backups (snapshots). We do not interact with etcd snapshots in the Rancher Backup tool or our code base here. So we do not use or interact with it during issue triage processes.

As mentioned, it would be helpful if you could provide us with the additional context requested in these questions:

• When you tried this out, did the FleetWorkspace you created get backed up (is it in the backup file)?
• If yes, did an error occur during the restore because of that resource that caused it to fail restoring other resources?
• If yes, what other resources specifically failed to restore due to the FleetWorkspace failing to restore?

Having this context would be very helpful to better understand the issue you're reporting her and the root cause behind it. Our team is very interested in helping sort this out whether this is caused by a bug in Rancher Backups or potentially somewhere in Fleet. First we will need a better understanding of the report here to identify the root cause.

---

As @jbiers mentioned, in Rancher Backups we already capture FleetWorkspace resources and upon restore the Fleet controller seems to correctly redeploy a matching Namespace for the FleetWorkspace. The wording of this case seemed to imply that was the root issue - however I believe I see what you're intending to report now and would like you to confirm it.

Instead of the issue actually being with FleetWorkspace restores - rather the issue is that to restore the ClusterGroup the Namespace it lives in needs to exist first. However, if we restore a FleetWorkspace and rely on fleet controller to recreate the Namespace, then it's possible we could attempt to restore a ClusterGroup into a Namespace that doesn't yet exist? Is this getting warmer or colder?

If this is warmer, then that's where the additional context I requested in those points above could have served to clarify. If we could review the logs from the Rancher Backup container during the restore, then it's possible the sequence of events and root error would be more obvious. Granted - if I've reached the correct understanding now those logs may not be necessary.

[jbiers]

@Daemonslayer2048 I have managed to reproduce your issue, or at least partially. Here are the steps I followed and the observed behavior

• Spin up a cluster with Rancher and the Backup tool installed.
• Create a FleetWorkspace in it of name enterprise, which automatically creates the enterprise namespace.

apiVersion: management.cattle.io/v3
kind: FleetWorkspace
metadata:
  name: enterprise

• Create a ClusterGroup in the enterprise namespace.

apiVersion: fleet.cattle.io/v1alpha1
kind: ClusterGroup
metadata:
  name: infra
  namespace: enterprise
spec:
  selector:
    matchExpressions: []
    matchLabels:
      infra: "true"

• Take a backup.
• Restore the backup in a new cluster.

The observed behavior in this case was indeed a failure in the Backup resource: error restoring namespaced resources, check logs for exact error. However, the Backup resource automatically starts out a retry process, in which the previously deployed resources are kept and only the failed ones are retried. As such, creating the ClusterGroup ends up being successful with the namespace now created, it just takes a while longer.

What we suppose could be happening in your case is that in a production cluster with a lot to be restored, the retry process could indeed take a while, and it might be a matter of waiting longer until it is completed. Can you try and see if you observe the same behavior in a lighter sandbox cluster?

Also, if you are a paying Rancher subscriber then a case could be opened with Rancher Support where they can formally file an issue and securely collect logs and debug data.