[BUG] Client certificates generated by kubeadm expire after 1 year

[rguske]

Describe the bug
By default, client certificates generated by kubeadm expire after 1 year.

See also Certificate Management with kubeadm.

Normally, certificates will be renewed during a Kubernetes version upgrade. Since upgrading to a new VEBA version (including PhotonOS, Kubernetes, Knative, etc.) is most likely in all cases done by just deploying a complete new instance, users will face this issue when running VEBA for more than 365 days.

We need to have this mentioned in our documentation.

To Reproduce

Check certificate expiration dates using kubeadm:

kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jun 15, 2024 13:27 UTC   262d            ca                      no      
apiserver                  Jun 15, 2024 13:27 UTC   262d            ca                      no      
apiserver-etcd-client      Jun 15, 2024 13:27 UTC   262d            etcd-ca                 no      
apiserver-kubelet-client   Jun 15, 2024 13:27 UTC   262d            ca                      no      
controller-manager.conf    Jun 15, 2024 13:27 UTC   262d            ca                      no      
etcd-healthcheck-client    Jun 15, 2024 13:27 UTC   262d            etcd-ca                 no      
etcd-peer                  Jun 15, 2024 13:27 UTC   262d            etcd-ca                 no      
etcd-server                Jun 15, 2024 13:27 UTC   262d            etcd-ca                 no      
front-proxy-client         Jun 15, 2024 13:27 UTC   262d            front-proxy-ca          no      
scheduler.conf             Jun 15, 2024 13:27 UTC   262d            ca                      no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jun 13, 2033 13:27 UTC   9y              no      
etcd-ca                 Jun 13, 2033 13:27 UTC   9y              no      
front-proxy-ca          Jun 13, 2033 13:27 UTC   9y              no

Expected behavior
It is actually an expected behavior of Kubernetes.

It's possible to renew the certificates manually using kubeadm certs renew all. A reboot is necessary after the renewal. Also, the update of the .kube/config file.

cp /root/.kube/config /root/.kube/.old-$(date --iso)-config
cp /etc/kubernetes/admin.conf /root/.kube/config

Screenshots
If applicable, add screenshots to help explain your problem.

Version (please complete the following information):

• VEBA Appliance
• Affects all versions

Additional context
Issue raised by the VEBA Slack community: Link to the thread.

[lamw]

Just had a user report that k8s certificate replacement works BUT the ingress endpoints don't work and this is because we also have ingress certs https://github.com/vmware-samples/vcenter-event-broker-appliance/blob/development/files/setup-09-ingress.sh#L43 that also need to get re-generated by deleting and re-creating the secret

[iwikus]

Hi, I am the mentioned user, I was able to get kubectl working using kubeadm certs renew all, but maybe it is not enough. I have custom function, setting vm custom attribute to username, which accessed vm console.
Cheking logs of trigger dispatcher there are refused connections. So something is not working yet.

kubectl -n vmware-functions logs veba-pcli-attr-console-trigger-dispatcher-d8b9b9d6f-q57qw

{"level":"warn","ts":"2023-10-24T16:27:55.683Z","logger":"rabbitmq-dispatcher","caller":"dispatcher/dispatcher.go:129","msg":"Invalid result type, not HTTP Result: Post "http://kn-pcli-attr-console.vmware-functions.svc.cluster.local\": dial tcp 10.101.179.242:80: connect: connection refused"}
{"level":"warn","ts":"2023-10-24T16:27:55.683Z","logger":"rabbitmq-dispatcher","caller":"dispatcher/dispatcher.go:166","msg":"Failed to deliver to "http://kn-pcli-attr-console.vmware-functions.svc.cluster.local\" requeue: false"}

[rguske]

I've tested, validated and documented the required steps. It'll be part of the PR which takes care of #1074