Access dqlite

[xl204431]

Background

a pod can be get under namespace, and the status stick as "Terminating"

but when get by name, describe or delete this pod , it shows that the pod can not be found.
Error from server (NotFound): pods "XXX" not found

Support need

I guess the data is corrupted and want to investigate dqlite's data.
How can I access dqlite?

[neoaggelos]

Hi @xl204431

To get a raw sqlite shell you can use the following command:

/snap/microk8s/current/bin/dqlite \
  --cert /var/snap/microk8s/current/var/kubernetes/backend/cluster.crt \
  --key /var/snap/microk8s/current/var/kubernetes/backend/cluster.key \
  --servers file:////var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml \
  k8s

To retrieve a snapshot of the database (current status of all keys/values), you might the following command to be useful:

microk8s dbctl --debug backup

This will print all current database keys and create a tarball that you can inspect. Hope this helps

[neoaggelos]

It would also help if you share some details about what is the issue that are you experiencing. Thanks again!

[xl204431]

@neoaggelos
Thank you so much for your response!! I think I might need more help!

Issue

I can view the pod when get namespace

root@somehost# microk8s.kubectl get pod -n some-namespace
NAME                        READY   STATUS    RESTARTS       AGE
problem-pod              1/1     Terminating   0              447d
normal-pod                1/1     Running       0              xxd
...
normal-pod                 1/1     Running       0              xxd

But I can not get, describe or delete this problem pod

root@somehost# microk8s.kubectl get pod problem-pod -n some-namespace
Error from server (NotFound): pods "problem-pod" not found
root@somehost# microk8s.kubectl describe pod problem-pod -n some-namespace
Error from server (NotFound): pods "problem-pod" not found
root@somehost# microk8s.kubectl delete pod problem-pod -n some-namespace
Error from server (NotFound): pods "problem-pod" not found

So, this problem-pod keeps its data and affect velero node-agent

Reproduce

Have no idea how it happened

Try to debug

I use microk8s dbctl --debug backup, and got some data, but some data can be displayed in text format, others are binary format. Why? How can I inspect this data?

I tried

/snap/microk8s/current/bin/dqlite \
  --cert /var/snap/microk8s/current/var/kubernetes/backend/cluster.crt \
  --key /var/snap/microk8s/current/var/kubernetes/backend/cluster.key \
  --servers file:////var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml \
  k8s

It connected, but I want to inspect what tables are in the db for further check, however, the command turns out to be error

dqlite> .tables
Error:  exec: near ".": syntax error

I try to learn k8s-dqlite's code, and only find migrator without any other tools to use.

I don't know how to solve this issue and need your help.

[neoaggelos]

Hi @xl204431, sorry for missing this

One way to observe the current state of the datastore is to use:

microk8s dbctl --debug backup

Example output from an empty cluster:

Backing up the datastore
INFO[0000] mode: backup-dqlite, endpoint: unix:///var/snap/microk8s/x1/var/kubernetes/backend/kine.sock:12379, dir: /tmp/tmpgon0wlxj/backup-2023-10-16-17-35-11 
DEBU[0000] 0) /registry/health                          
DEBU[0000] 1) /registry/ranges/servicenodeports         
DEBU[0000] 2) /registry/apiregistration.k8s.io/apiservices/v1.apps 
DEBU[0000] 3) /registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io 
DEBU[0000] 4) /registry/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io 
DEBU[0000] 5) /registry/apiregistration.k8s.io/apiservices/v1. 
DEBU[0000] 6) /registry/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io 
DEBU[0000] 7) /registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io 
DEBU[0000] 8) /registry/namespaces/kube-system          
DEBU[0000] 9) /registry/apiregistration.k8s.io/apiservices/v1.autoscaling 
DEBU[0000] 10) /registry/apiregistration.k8s.io/apiservices/v2.autoscaling 
DEBU[0000] 11) /registry/apiregistration.k8s.io/apiservices/v1.batch 
DEBU[0000] 12) /registry/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io 
DEBU[0000] 13) /registry/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io 
DEBU[0000] 14) /registry/namespaces/kube-public         
DEBU[0000] 15) /registry/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io 
DEBU[0000] 16) /registry/configmaps/kube-system/extension-apiserver-authentication 
DEBU[0000] 17) /registry/apiregistration.k8s.io/apiservices/v1.events.k8s.io 
DEBU[0000] 18) /registry/configmaps/kube-system/kube-apiserver-legacy-service-account-token-tracking 
DEBU[0000] 19) /registry/prioritylevelconfigurations/system 
DEBU[0000] 20) /registry/apiregistration.k8s.io/apiservices/v1beta2.flowcontrol.apiserver.k8s.io 
DEBU[0000] 21) /registry/apiregistration.k8s.io/apiservices/v1beta3.flowcontrol.apiserver.k8s.io 
DEBU[0000] 22) /registry/apiregistration.k8s.io/apiservices/v1.networking.k8s.io 
DEBU[0000] 23) /registry/namespaces/kube-node-lease     
DEBU[0000] 24) /registry/apiregistration.k8s.io/apiservices/v1.node.k8s.io 
DEBU[0000] 25) /registry/apiregistration.k8s.io/apiservices/v1.policy 
DEBU[0000] 26) /registry/prioritylevelconfigurations/node-high 
DEBU[0000] 27) /registry/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io 
DEBU[0000] 28) /registry/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io 
DEBU[0000] 29) /registry/apiregistration.k8s.io/apiservices/v1.storage.k8s.io 
DEBU[0000] 30) /registry/namespaces/default             
DEBU[0000] 31) /registry/prioritylevelconfigurations/leader-election 
DEBU[0000] 32) /registry/prioritylevelconfigurations/workload-high 
DEBU[0000] 33) /registry/prioritylevelconfigurations/workload-low 
DEBU[0000] 34) /registry/prioritylevelconfigurations/global-default 
DEBU[0000] 35) /registry/flowschemas/system-nodes       
DEBU[0000] 36) /registry/flowschemas/system-node-high   
DEBU[0000] 37) /registry/flowschemas/system-leader-election 
DEBU[0000] 38) /registry/flowschemas/workload-leader-election 
DEBU[0000] 39) /registry/flowschemas/endpoint-controller 
DEBU[0000] 40) /registry/flowschemas/kube-controller-manager 
DEBU[0000] 41) /registry/flowschemas/kube-scheduler     
DEBU[0000] 42) /registry/prioritylevelconfigurations/catch-all 
DEBU[0000] 43) /registry/flowschemas/service-accounts   
DEBU[0000] 44) /registry/prioritylevelconfigurations/exempt 
DEBU[0000] 45) /registry/flowschemas/kube-system-service-accounts 
DEBU[0000] 46) /registry/flowschemas/probes             
DEBU[0000] 47) /registry/flowschemas/global-default     
DEBU[0000] 48) /registry/flowschemas/exempt             
DEBU[0000] 49) /registry/flowschemas/catch-all          
DEBU[0000] 50) /registry/priorityclasses/system-node-critical 
DEBU[0000] 51) /registry/priorityclasses/system-cluster-critical 
DEBU[0000] 52) /registry/services/specs/default/kubernetes 
DEBU[0000] 53) /registry/services/endpoints/default/kubernetes 
DEBU[0000] 54) /registry/endpointslices/default/kubernetes 
DEBU[0000] 55) /registry/csinodes/test-core-22          
DEBU[0000] 56) /registry/serviceaccounts/kube-system/calico-kube-controllers 
DEBU[0000] 57) /registry/serviceaccounts/kube-system/calico-node 
DEBU[0000] 58) /registry/configmaps/kube-system/calico-config 
DEBU[0000] 59) /registry/apiregistration.k8s.io/apiservices/v1.crd.projectcalico.org 
DEBU[0000] 60) /registry/apiextensions.k8s.io/customresourcedefinitions/bgpconfigurations.crd.projectcalico.org 
DEBU[0000] 61) /registry/apiextensions.k8s.io/customresourcedefinitions/bgppeers.crd.projectcalico.org 
DEBU[0000] 62) /registry/apiextensions.k8s.io/customresourcedefinitions/blockaffinities.crd.projectcalico.org 
DEBU[0000] 63) /registry/apiextensions.k8s.io/customresourcedefinitions/caliconodestatuses.crd.projectcalico.org 
DEBU[0000] 64) /registry/apiextensions.k8s.io/customresourcedefinitions/clusterinformations.crd.projectcalico.org 
DEBU[0000] 65) /registry/apiextensions.k8s.io/customresourcedefinitions/felixconfigurations.crd.projectcalico.org 
DEBU[0000] 66) /registry/apiextensions.k8s.io/customresourcedefinitions/globalnetworkpolicies.crd.projectcalico.org 
DEBU[0000] 67) /registry/apiextensions.k8s.io/customresourcedefinitions/globalnetworksets.crd.projectcalico.org 
DEBU[0000] 68) /registry/apiextensions.k8s.io/customresourcedefinitions/hostendpoints.crd.projectcalico.org 
DEBU[0000] 69) /registry/apiextensions.k8s.io/customresourcedefinitions/ipamblocks.crd.projectcalico.org 
DEBU[0000] 70) /registry/apiextensions.k8s.io/customresourcedefinitions/ipamconfigs.crd.projectcalico.org 
DEBU[0000] 71) /registry/apiextensions.k8s.io/customresourcedefinitions/ipamhandles.crd.projectcalico.org 
DEBU[0000] 72) /registry/apiextensions.k8s.io/customresourcedefinitions/ippools.crd.projectcalico.org 
DEBU[0000] 73) /registry/apiextensions.k8s.io/customresourcedefinitions/ipreservations.crd.projectcalico.org 
DEBU[0000] 74) /registry/apiextensions.k8s.io/customresourcedefinitions/kubecontrollersconfigurations.crd.projectcalico.org 
DEBU[0000] 75) /registry/clusterroles/calico-kube-controllers 
DEBU[0000] 76) /registry/apiextensions.k8s.io/customresourcedefinitions/networkpolicies.crd.projectcalico.org 
DEBU[0000] 77) /registry/clusterroles/calico-node       
DEBU[0000] 78) /registry/apiextensions.k8s.io/customresourcedefinitions/networksets.crd.projectcalico.org 
DEBU[0000] 79) /registry/clusterrolebindings/calico-kube-controllers 
DEBU[0000] 80) /registry/clusterrolebindings/calico-node 
DEBU[0000] 81) /registry/configmaps/kube-public/local-registry-hosting 
DEBU[0000] 82) /registry/serviceaccounts/kube-system/coredns 
DEBU[0000] 83) /registry/configmaps/kube-system/coredns 
DEBU[0000] 84) /registry/ranges/serviceips              
DEBU[0000] 85) /registry/services/specs/kube-system/kube-dns 
DEBU[0000] 86) /registry/clusterroles/coredns           
DEBU[0000] 87) /registry/clusterrolebindings/coredns    
DEBU[0000] 88) /registry/serviceaccounts/kube-system/endpoint-controller 
DEBU[0000] 89) /registry/serviceaccounts/kube-system/generic-garbage-collector 
DEBU[0000] 90) /registry/serviceaccounts/kube-system/clusterrole-aggregation-controller 
DEBU[0000] 91) /registry/serviceaccounts/kube-system/endpointslicemirroring-controller 
DEBU[0000] 92) /registry/serviceaccounts/kube-system/replication-controller 
DEBU[0000] 93) /registry/serviceaccounts/kube-system/statefulset-controller 
DEBU[0000] 94) /registry/serviceaccounts/kube-system/certificate-controller 
DEBU[0000] 95) /registry/serviceaccounts/kube-system/endpointslice-controller 
DEBU[0000] 96) /registry/serviceaccounts/kube-system/job-controller 
DEBU[0000] 97) /registry/serviceaccounts/kube-system/disruption-controller 
DEBU[0000] 98) /registry/serviceaccounts/kube-system/cronjob-controller 
DEBU[0000] 99) /registry/serviceaccounts/kube-system/ttl-controller 
DEBU[0000] 100) /registry/serviceaccounts/kube-system/ephemeral-volume-controller 
DEBU[0000] 101) /registry/serviceaccounts/kube-system/node-controller 
DEBU[0000] 102) /registry/serviceaccounts/kube-system/service-controller 
DEBU[0000] 103) /registry/serviceaccounts/kube-system/ttl-after-finished-controller 
DEBU[0000] 104) /registry/serviceaccounts/kube-system/pod-garbage-collector 
DEBU[0000] 105) /registry/serviceaccounts/kube-system/resourcequota-controller 
DEBU[0000] 106) /registry/serviceaccounts/kube-system/daemon-set-controller 
DEBU[0000] 107) /registry/serviceaccounts/kube-system/deployment-controller 
DEBU[0000] 108) /registry/serviceaccounts/kube-system/replicaset-controller 
DEBU[0000] 109) /registry/serviceaccounts/kube-system/attachdetach-controller 
DEBU[0000] 110) /registry/serviceaccounts/kube-system/expand-controller 
DEBU[0000] 111) /registry/serviceaccounts/kube-system/horizontal-pod-autoscaler 
DEBU[0000] 112) /registry/serviceaccounts/kube-system/pv-protection-controller 
DEBU[0000] 113) /registry/serviceaccounts/kube-system/namespace-controller 
DEBU[0000] 114) /registry/serviceaccounts/kube-system/service-account-controller 
DEBU[0000] 115) /registry/serviceaccounts/kube-system/persistent-volume-binder 
DEBU[0000] 116) /registry/serviceaccounts/kube-system/pvc-protection-controller 
DEBU[0000] 117) /registry/serviceaccounts/kube-system/root-ca-cert-publisher 
DEBU[0000] 118) /registry/configmaps/default/kube-root-ca.crt 
DEBU[0000] 119) /registry/configmaps/kube-node-lease/kube-root-ca.crt 
DEBU[0000] 120) /registry/configmaps/kube-public/kube-root-ca.crt 
DEBU[0000] 121) /registry/configmaps/kube-system/kube-root-ca.crt 
DEBU[0000] 122) /registry/serviceaccounts/default/default 
DEBU[0000] 123) /registry/serviceaccounts/kube-node-lease/default 
DEBU[0000] 124) /registry/serviceaccounts/kube-public/default 
DEBU[0000] 125) /registry/serviceaccounts/kube-system/default 
DEBU[0000] 126) /registry/controllerrevisions/kube-system/calico-node-6fbb45588b 
DEBU[0000] 127) /registry/crd.projectcalico.org/ippools/default-ipv4-ippool 
DEBU[0000] 128) /registry/crd.projectcalico.org/clusterinformations/default 
DEBU[0000] 129) /registry/crd.projectcalico.org/felixconfigurations/default 
DEBU[0000] 130) /registry/crd.projectcalico.org/ipamconfigs/default 
DEBU[0000] 131) /registry/crd.projectcalico.org/blockaffinities/test-core-22-10-1-19-0-26 
DEBU[0000] 132) /registry/crd.projectcalico.org/ipamhandles/vxlan-tunnel-addr-test-core-22 
DEBU[0000] 133) /registry/replicasets/kube-system/coredns-864597b5fd 
DEBU[0000] 134) /registry/deployments/kube-system/coredns 
DEBU[0000] 135) /registry/crd.projectcalico.org/kubecontrollersconfigurations/default 
DEBU[0000] 136) /registry/crd.projectcalico.org/ipamhandles/k8s-pod-network.ec2465275b1b20754b2a53332dd46835c769a24a343e42e40cc606ffb2444ede 
DEBU[0000] 137) /registry/pods/kube-system/coredns-864597b5fd-q6bcc 
DEBU[0000] 138) /registry/endpointslices/kube-system/kube-dns-c5z7m 
DEBU[0000] 139) /registry/controllerrevisions/kube-system/calico-node-fd77f9c5 
DEBU[0000] 140) /registry/services/endpoints/kube-system/kube-dns 
DEBU[0000] 141) /registry/replicasets/kube-system/calico-kube-controllers-77bd7c5b 
DEBU[0000] 142) /registry/crd.projectcalico.org/ipamhandles/k8s-pod-network.7cdb6ae634d6eb77dddf5ded3c9a729cfa3c4011e5d937d930b0fcf2d2361f5a 
DEBU[0000] 143) /registry/crd.projectcalico.org/ipamblocks/10-1-19-0-26 
DEBU[0000] 144) /registry/pods/kube-system/calico-node-cg6nq 
DEBU[0000] 145) /registry/daemonsets/kube-system/calico-node 
DEBU[0000] 146) /registry/pods/kube-system/calico-kube-controllers-66d4dc5c95-8wqt5 
DEBU[0000] 147) /registry/replicasets/kube-system/calico-kube-controllers-66d4dc5c95 
DEBU[0000] 148) /registry/poddisruptionbudgets/kube-system/calico-kube-controllers 
DEBU[0000] 149) /registry/deployments/kube-system/calico-kube-controllers 
DEBU[0000] 150) /registry/minions/test-core-22          
DEBU[0000] 151) /registry/leases/kube-node-lease/test-core-22 
DEBU[0000] 152) /registry/leases/kube-system/apiserver-j3o5tsgk4d6wtav6cqoosazzvi 
DEBU[0000] 153) /registry/leases/kube-system/kube-controller-manager 
DEBU[0000] 154) /registry/masterleases/172.16.101.59    
DEBU[0000] 155) /registry/leases/kube-system/kube-scheduler 
The backup is: backup-2023-10-16-17-35-11.tar.gz

The debug output shows the index of each key in the datastore. You can then inspect the tarball to check the $index.key and $index.value files. The first should be the key, the other should be the contents (most likely raw protobuf bytes)

Hope this helps

[xl204431]

@neoaggelos
I tried this. The problem is that I don't know $index.value file's format thus I cannot recognize its data. Are there any tools to inspect this data file?

[neoaggelos]

Hi @xl204431

I imagine this is using the internal protobufs found on the Kubernetes project, so I am not sure if I can help more in this direction. The type definition would be https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/core/v1/types.go#L4333

I am not aware of any tooling to directly inspect the data.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.