Container Registry for K8S Cluster

[haminhcong]

Goals

• Create a container registry
• Auto config k8s cluster node to use registry:
  • Set ssl cert config
  • Or set insecure registry
• Allow external user pull and push to container registry

Environment

• Kubepsray 2.15
• containerd 1.4.x
• Registry Storage: S3 like: Ceph, Minio - Provide outside cluster

Problem 1: Install Container Registry

Tools

• Harbor: https://github.com/goharbor/harbor

Install methods:

• Install outside cluster: Harbor install on three VMs outside K8S Cluster, then configure containerd config berfore install cluster
• Install inside cluster: Harbor install on K8S Cluster, then reconfigure containerd configuration of running Cluster. Recheck if need reconfigure container engine after install Harbor or not ?

Problem 2: Configure cluster node registry certificate

To resolve problem x509: certificate signed by unknown authority we have options

Solution 1 - config containerd using registry ca cert before start containerd

[plugins.cri.registry.configs."my.custom.registry".tls]
    ca_file   = "/etc/containerd/my.custom.registry/ca.crt"
    cert_file = "/etc/containerd/my.custom.registry/cert.pem"
    key_file  = "/etc/containerd/my.custom.registry/key.pem"

Solution 2 - insecure registry

[plugins.cri.registry.configs."my.custom.registry".tls]
  insecure_skip_verify = true

Solution 3 - using registry ca cert in system cert

Follow issue containerd/containerd#3071 we need put ca cert to folder /usr/local/share/ca-certificates before start containerd service, or after put ca cert we need restart containerd service.

https://github.com/containerd/cri/blob/release/1.4/docs/registry.md#configure-registry-tls-communication

References

• https://github.com/containerd/cri/blob/release/1.4/docs/registry.md#configure-registry-tls-communication
• Create a way to provide the CA of a self signed certificate to containerd without restarting it containerd/containerd#3071
• https://www.ovh.com/blog/managing-harbor-at-cloud-scale-the-story-behind-harbor-kubernetes-operator/
• https://www.cisco.com/c/en/us/td/docs/net_mgmt/cisco_container_platform/7-0/User_Guide/ccp-user-guide-7-0-0/CCP-User-Guide-6-0-0_chapter_01010.pdf