CRC installation / restart keeps failing

[kai-mersmann-ecc]

Dear CRC community,
we have been using CRC sucessfully on several workstations on top of ubuntu 22.04 for more than 1,5 years now. About 8 weeks ago we saw CRC restart and installations failing - and could not find the cause for this. We tried different CRC version

CRC Version 2.15.0 with openshift 4.12.5
CRC Version 2.6.0 with openshift 4.10.22
CRC Version 2.22.1 with openshift 4.13.3
CRC Version 2.20.0 with openshift 4.13.0

and also checked routing, enabled/disabled VPN and Proxy and tried everything else. Yet we could not finde the root cause and even the erros we seer are inconsistent.

We use the following config:

 crc config view
- bundle                                : /home/<homedir>/ecc_local_repo/crc_libvirt_4.13.3_amd64.crcbundle
- consent-telemetry                     : no
- cpus                                  : 8
- disk-size                             : 120
- memory                                : 30720
- pull-secret-file                      : /home/<homedir>/pull-secret
- skip-check-systemd-networkd-running   : true

In about 1/3 of installations, initial install is possible, we can login into the console and oc login works as expected. But after crc stop && crc start, we see receive the same errors as described in the next section

Yet in about 2/3 of the installation, the initial installation fails.
While even the operators that fail are not consistent, most of the time we receive something like
2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver" ... > And eventually we run into a timeout.

Repeated restarts to not improve the situation.

I followed the instructions in https://github.com/crc-org/crc/wiki/Debugging-guide, yet could not find any root-cause. Some operators are degraded, but listing the pods/logs/events in the respective namespace showed many different errors, yet I could not conclude anything from it.

KUBECONFIG=/opt/kubeconfig kubectl get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.13.3    False       False         True       37m     APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.oauth.openshift.io: not available: failing or missing response from https://10.217.0.7:8443/apis/oauth.openshift.io/v1: Get "https://10.217.0.7:8443/apis/oauth.openshift.io/v1": dial tcp 10.217.0.7:8443: connect: connection refused...
config-operator                            4.13.3    True        False         False      23d     
console                                    4.13.3    True        False         False      68m     
control-plane-machine-set                  4.13.3    True        False         False      23d     
dns                                        4.13.3    True        False         False      73m     
etcd                                       4.13.3    True        False         False      23d     
image-registry                             4.13.3    True        False         False      68m     
ingress                                    4.13.3    True        False         False      23d     
kube-apiserver                             4.13.3    True        False         False      23d     
kube-controller-manager                    4.13.3    True        False         True       23d     StaticPodsDegraded: pod/kube-controller-manager-crc-ccssl-master-0 container "cluster-policy-controller" is waiting: CrashLoopBackOff: back-off 5m0s restarting failed container=cluster-policy-controller pod=kube-controller-manager-crc-ccssl-master-0_openshift-kube-controller-manager(209cd685a470aca527b87e5e91df0c1c)
kube-scheduler                             4.13.3    True        False         False      23d     
kube-storage-version-migrator              4.13.3    True        False         False      73m     
machine-api                                4.13.3    True        False         False      23d     
machine-approver                           4.13.3    True        False         False      23d     
machine-config                             4.13.3    False       False         True       27m     Cluster not available for [{operator 4.13.3}]: failed to apply machine config daemon manifests: error during waitForDaemonsetRollout: [timed out waiting for the condition, daemonset machine-config-daemon is not ready. status: (desired: 1, updated: 1, ready: 0, unavailable: 1)]
marketplace                                4.13.3    True        False         False      23d     
network                                    4.13.3    True        False         False      23d     
openshift-apiserver                        4.13.3    False       False         True       49m     APIServerDeploymentAvailable: no apiserver.openshift-apiserver pods available on any node....
openshift-controller-manager               4.13.3    False       True          False      45m     Available: no openshift controller manager daemon pods available on any node.
openshift-samples                          4.13.3    True        False         False      23d     
operator-lifecycle-manager                 4.13.3    True        False         False      23d     
operator-lifecycle-manager-catalog         4.13.3    True        False         False      23d     
operator-lifecycle-manager-packageserver   4.13.3    False       True          False      53m     ClusterServiceVersion openshift-operator-lifecycle-manager/packageserver observed in phase Failed with reason: InstallCheckFailed, message: install timeout
service-ca                                 4.13.3    True        False         False      23d     

Please see the output of the installation and a following restart below

crc config view[/home/kaimersmann/git/exg.tools.ecc.lde/.diagnostics/ticket/crc_afterinstallation.log]
- bundle                                : /home/<myhomedir>/ecc_local_repo/crc_libvirt_4.13.3_amd64.crcbundle
- consent-telemetry                     : no
- cpus                                  : 8
- disk-size                             : 120
- memory                                : 30720
- pull-secret-file                      : /home/<myhomerdir>/pull-secret
- skip-check-systemd-networkd-running   : true

INFO Using bundle path /home/kaimersmann/ecc_local_repo/crc_libvirt_4.13.3_amd64.crcbundle 
INFO Checking if running as non-root              
INFO Checking if running inside WSL2              
INFO Checking if crc-admin-helper executable is cached 
INFO Caching crc-admin-helper executable          
INFO Using root access: Changing ownership of /home/kaimersmann/.crc/bin/crc-admin-helper-linux 
INFO Using root access: Setting suid for /home/kaimersmann/.crc/bin/crc-admin-helper-linux 
INFO Checking if running on a supported CPU architecture 
INFO Checking if crc executable symlink exists    
INFO Creating symlink for crc executable          
INFO Checking minimum RAM requirements            
INFO Checking if Virtualization is enabled        
INFO Checking if KVM is enabled                   
INFO Checking if libvirt is installed             
INFO Checking if user is part of libvirt group    
INFO Checking if active user/process is currently part of the libvirt group 
INFO Checking if libvirt daemon is running        
INFO Checking if a supported libvirt version is installed 
INFO Checking if crc-driver-libvirt is installed  
INFO Installing crc-driver-libvirt                
INFO Checking crc daemon systemd service          
INFO Setting up crc daemon systemd service        
INFO Checking crc daemon systemd socket units     
INFO Setting up crc daemon systemd socket units   
INFO Checking if AppArmor is configured           
INFO Updating AppArmor configuration              
INFO Using root access: Updating AppArmor configuration 
INFO Using root access: Changing permissions for /etc/apparmor.d/libvirt/TEMPLATE.qemu to 644  
INFO Checking if systemd-networkd is running      
WARN Skipping above check...                      
INFO Checking if NetworkManager is installed      
INFO Checking if NetworkManager service is running 
INFO Checking if dnsmasq configurations file exist for NetworkManager 
INFO Checking if the systemd-resolved service is running 
INFO Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists 
INFO Writing NetworkManager dispatcher file for crc 
INFO Using root access: Writing NetworkManager configuration to /etc/NetworkManager/dispatcher.d/99-crc.sh 
INFO Using root access: Changing permissions for /etc/NetworkManager/dispatcher.d/99-crc.sh to 755  
INFO Using root access: Executing systemctl daemon-reload command 
INFO Using root access: Executing systemctl reload NetworkManager 
INFO Checking if libvirt 'crc' network is available 
INFO Setting up libvirt 'crc' network             
INFO Checking if libvirt 'crc' network is active  
INFO Starting libvirt 'crc' network               
INFO Checking if CRC bundle is extracted in '$HOME/.crc' 
INFO Checking if /home/kaimersmann/ecc_local_repo/crc_libvirt_4.13.3_amd64.crcbundle exists 
INFO Getting bundle for the CRC executable        
INFO Downloading bundle: /home/kaimersmann/ecc_local_repo/crc_libvirt_4.13.3_amd64.crcbundle... 
INFO Uncompressing /home/kaimersmann/ecc_local_repo/crc_libvirt_4.13.3_amd64.crcbundle 
crc.qcow2:  14.24 GiB / 14.24 GiB [-----------------------------------------------------------------------------------------------------] 100.00%
oc:  136.05 MiB / 136.05 MiB [----------------------------------------------------------------------------------------------------------] 100.00%
Your system is correctly setup for using CRC. Use 'crc start' to start the instance



& route 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         _gateway        0.0.0.0         UG    100    0        0 enx3c18a0177983
link-local      0.0.0.0         255.255.0.0     U     1000   0        0 crc
192.168.0.0     0.0.0.0         255.255.255.0   U     100    0        0 enx3c18a0177983
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
192.168.130.0   0.0.0.0         255.255.255.0   U     0      0        0 crc


### No Proxy ist set



&crc start

.
.
.
time="2023-07-13T17:51:11+02:00" level=info msg="Operators are stable (3/3)..."
time="2023-07-13T17:51:11+02:00" level=debug msg="Cluster took 7m0.313904251s to stabilize"
time="2023-07-13T17:51:11+02:00" level=info msg="Adding crc-admin and crc-developer contexts to kubeconfig..."


# All controller came up - no errors
Openshift-Console working as expected




#Login successful.

You have access to 65 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default"

On this ocassion, everything looks fine.

CONTAINER           IMAGE                                                                                                                        CREATED              STATE               NAME                                          ATTEMPT             POD ID              POD
fa26ebdd87662       af5a03bdd6e026ea713ac7fd8d1944b19a65f83c27a7a2922456f134616ee24a                                                             About a minute ago   Running             oauth-openshift                               0                   022c7e129b535       oauth-openshift-6845c4498f-dc7vn
eac13c8c3e7e7       06a2691e725cedcc91a21b726261737858120a28bb95d36c6be33bd2b0acbfb4                                                             5 minutes ago        Running             registry                                      0                   47928c76cce35       image-registry-6c86d7b644-6c47r
3da4baa06f9ba       a6841cf5cc0eeb0c43d3018d6b1e108a15bc51914f9e9a9d7c5060fb7a56897f                                                             6 minutes ago        Running             route-controller-manager                      0                   754af16d6c3e6       route-controller-manager-cc456787c-jvkd4
18dc07099c8a5       a6b3a54ce77a70570298864a72dfa328ca779803eae9a62a017c6755e7c6b00f                                                             6 minutes ago        Running             controller-manager                            0                   a9b18863aed49       controller-manager-75dc6bbf85-2mdt8
96e540c3fedda       registry.redhat.io/redhat/community-operator-index@sha256:77069de3077f14d8876a8bd2af47b13cd6e55ec60c30b72392beb67e298d1bc3   7 minutes ago        Running             registry-server                               0                   3ce72cea5ffdf       community-operators-rjft7
1fab598c91fd5       registry.redhat.io/redhat/redhat-marketplace-index@sha256:3bb92b5f6504502e746e26a15db0a39d0842ad12253f039e89afde6934bbb64c   7 minutes ago        Running             registry-server                               0                   06e3c25e4ee8f       redhat-marketplace-z6kxv
71f2b5263e4cd       registry.redhat.io/redhat/certified-operator-index@sha256:5b318f556679c031074b2cf8e93d8a2c83aa95289b52d4da994d9a8a36abf951   7 minutes ago        Running             registry-server                               0                   cba14c29d5658       certified-operators-7mjj2
4d2e01ae3f908       registry.redhat.io/redhat/redhat-operator-index@sha256:1fd9adddcf940d836aa88234a091d95ff21c6be122b934775c6febf596c8b54f      7 minutes ago        Running             registry-server                               0                   30557ed9775a4       redhat-operators-r6zjp
06207431303c0       781a37f1fe2fb4e5b68871e955412aa312fc7a377fe8cecdee85e7d9276bb1bc                                                             9 minutes ago        Running             ingress-operator                              1                   ed56a2a1037c3       ingress-operator-5fd5cc48f8-dtl8m


$ crc stop
INFO Stopping kubelet and all containers...       
INFO Stopping the instance, this may take a few minutes... 
Stopped the instance


$ crc start
WARN A new version (2.23.0) has been published on https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/2.23.0/crc-linux-amd64.tar.xz 
INFO Checking if running as non-root              
INFO Checking if running inside WSL2              
INFO Checking if crc-admin-helper executable is cached 
INFO Checking if running on a supported CPU architecture 
INFO Checking if crc executable symlink exists    
INFO Checking minimum RAM requirements            
INFO Checking if Virtualization is enabled        
INFO Checking if KVM is enabled                   
INFO Checking if libvirt is installed             
INFO Checking if user is part of libvirt group    
INFO Checking if active user/process is currently part of the libvirt group 
INFO Checking if libvirt daemon is running        
INFO Checking if a supported libvirt version is installed 
INFO Checking if crc-driver-libvirt is installed  
INFO Checking crc daemon systemd socket units     
INFO Checking if AppArmor is configured           
INFO Checking if systemd-networkd is running      
WARN Skipping above check...                      
INFO Checking if NetworkManager is installed      
INFO Checking if NetworkManager service is running 
INFO Checking if dnsmasq configurations file exist for NetworkManager 
INFO Checking if the systemd-resolved service is running 
INFO Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists 
INFO Checking if libvirt 'crc' network is available 
INFO Checking if libvirt 'crc' network is active  
INFO Loading bundle: crc_libvirt_4.13.3_amd64...  
INFO Starting CRC VM for openshift 4.13.3...      [/home/kaimersmann/git/exg.tools.ecc.lde/.diagnostics/ticket/crc_2ndrestart.log](url)
INFO CRC instance is running with IP 192.168.130.11 
INFO CRC VM is running                            
INFO Updating authorized keys...                  
INFO Resizing /dev/vda4 filesystem                
INFO Configuring shared directories               
INFO Check internal and public DNS query...       
INFO Check DNS query from host...                 
WARN Wildcard DNS resolution for apps-crc.testing does not appear to be working 
INFO Verifying validity of the kubelet certificates... 
INFO Starting kubelet service                     
INFO Waiting for kube-apiserver availability... [takes around 2min] 
INFO Waiting until the user's pull secret is written to the instance disk... 
INFO Starting openshift instance... [waiting for the cluster to stabilize] 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
INFO 2 operators are progressing: openshift-controller-manager, operator-lifecycle-manager-packageserver 
ERRO Cluster is not ready: cluster operators are still not stable after 10m0.616896879s 
INFO Adding crc-admin and crc-developer contexts to kubeconfig... 
ERRO Cannot update kubeconfig: Head "https://oauth-openshift.apps-crc.testing": dial tcp 192.168.130.11:443: connect: connection refused 
Started the OpenShift cluster.

The server is accessible via web console at:
  https://console-openshift-console.apps-crc.testing

Log in as administrator:
  Username: kubeadmin
  Password: ******

Log in as user:
  Username: developer
  Password: developer

Use the 'oc' command line interface:
  $ eval $(crc oc-env)
  $ oc login -u developer https://api.crc.testing:6443

Anybody ran into this problem, or any advice how to further analyze?
Thanks in Advance&
Kind Regards

Kai

[cfergeau]

Dear CRC community, we have been using CRC sucessfully on several workstations on top of ubuntu 22.04 for more than 1,5 years now. About 8 weeks ago we saw CRC restart and installations failing - and could not find the cause for this. We tried different CRC version

CRC Version 2.15.0 with openshift 4.12.5 CRC Version 2.6.0 with openshift 4.10.22 CRC Version 2.22.1 with openshift 4.13.3 CRC Version 2.20.0 with openshift 4.13.0

Did some of these versions work flawlessly 2 months ago? Are there versions which were working before, and stopped working 8 weeks ago? Have you been using skip-check-systemd-networkd-running for a long time, or was this added recently?

ERRO Cluster is not ready: cluster operators are still not stable after 10m0.616896879s

This unfortunately can happen if the machine is a bit slow/... If you wait 10 more minutes and try KUBECONFIG=~/.crc/machines/crc/kubeconfig oc get co, do you still have some operators which are not ready? oc get co -ojson might have more details about why the operators are degraded.