Issue with standing up kubernetes after upgrading to 1.16+

[jaebchoi]

Actual Behavior

Once I make changes to Rancherr such as upgrade to 1.16 or upping RAM, I started to have issue with standing up kubernetes
I can get away with this issue If I am on specific VPN somehow.

There is workaround in which is using specific VPN or reset container image, completely uninstall Rancher Desktop and reinstall previous version ( 1.12.3) seems to do the trick.

Steps to Reproduce

Make any changes to Rancher Desktop (Upping RAM or upgrade to 1.16.0)
Start up any service that uses kubernetes in the Rancher Desktop.
Seems like kubernetes doesn't start on certain VPN.

Result

 'debug1: identity file /Users/namanrawal/Library/Application Support/rancher-desktop/lima/_config/user type 3\r\n' +
    'debug1: identity file /Users/namanrawal/Library/Application Support/rancher-desktop/lima/_config/user-cert type -1\r\n' +
    'debug1: Local version string SSH-2.0-OpenSSH_9.7\r\n' +
    'kex_exchange_identification: read: Connection reset by peer\r\n' +
    'Connection reset by 127.0.0.1 port 63303\r\n' +
    '/usr/bin/scp: Connection closed\r\n',
  code: 255,
  [Symbol(child-process.command)]: '/Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl.ventura --debug copy /var/folders/9v/3d0ljp391v9b5xt51bvsyf7m0000gp/T/rd-nerdctl-IOFC1eAfvY1Y/nerdctl 0:/tmp/rd-nerdctl-IOFC1eAfvY1Y.nerdctl'
}

Expected Behavior

Kubernetes can be stood up without any errors regardless of VPN

Additional Information

No response

Rancher Desktop Version

1.16.0

Rancher Desktop K8s Version

1.30.6

Which container engine are you using?

moby (dockercli)

What operating system are you using?

macOS

Operating System / Build Version

MacOS

What CPU architecture are you using?

x64

Linux only: what package format did you use to install Rancher Desktop?

None

Windows User Only

No response

[ewilkins-csi]

To add some more context to this, we've seen this for a number of people seemingly starting around the time of the 1.16 release, though this may be a red herring as the issue has happened after forcibly downgrading to 1.15 and 1.14 as well. All of the users with issues have had Macs, and either all or most of them have had ARM CPUs. (Haven't verified it's ARM in all cases, but in all cases I've checked it's been ARM).

It seems related to the CA cert import process that happens, though the symptoms vary. There's a Zscaler instance in place that is re-issuing certs and in most cases we've seen something like an x509 verification failure -- most often when fetching k8s versions.

Interestingly, in the workaround mentioned above, users can freely upgrade to 1.16 but they have to avoid the problematic VPN connection. If they use the VPN while Rancher Desktop is active and make any changes to their preferences, it gets into a bad state and requires either a fresh install or a factory reset. (Unclear at this point whether downgrading to 1.12 is necessary, though some users have said it doesn't work until they go back to that version.)

We did pull the logs from the cert import and there are definitely some certificates being skipped but it's unclear from the logs why they are being skipped.