helm chart tenant deployment v7.0.0 doesn't work for private images registry

[bono5150]

minio-operator/minio-operator 4.3.7 v4.3.7 A Helm chart for MinIO Operator

Helm chart values located here:
https://github.com/minio/operator/blob/master/helm/tenant/values.yaml

doesn't take under consideration parameter imagePullSecret: { } from line 39.

Expected Behavior

It is possible to create tenant from image stored in private registry.

Current Behavior

Tenant deployment constantly reports deployment errors:
Back-off pulling image "registry.home.lab/registry/minio:RELEASE.2024-11-07T00-52-20Z"
Error: ImagePullBackOff
Failed to pull image "registry.home.lab/registry/minio:RELEASE.2024-11-07T00-52-20Z": failed to pull and unpack image "registry.home.lab/registry/minio:RELEASE.2024-11-07T00-52-20Z": failed to resolve reference "registry.home.lab/registry/minio:RELEASE.2024-11-07T00-52-20Z": pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials
Readiness probe failed: HTTP probe failed with statuscode: 500

The same configuration for minio operator works perfectly fine (the values file contains imagePullSecrets parameter - be cautious the name differs from the tenant values imagePullSecret and the type is not a vector but a scalar)

Possible Solution

Clear documentation of usage.
imagePullSecret scalar parameter taken under consideration by Tenant helm chart

Steps to Reproduce (for bugs)

1. Create private images docker registry
2. Pull the quay.io/minio/minio:RELEASE.2024-11-07T00-52-20Z image to private registry
3. Pull the quay.io/minio/operator-sidecar:v7.0.0 image to private registry
4. Pull the quay.io/minio/operator:v7.0.0 image to private regsitry
5. Modify values for opeator and tenant to point to images from private regsitry (https://github.com/minio/operator/blob/master/helm/tenant/values.yaml, https://github.com/minio/operator/blob/master/helm/operator/values.yaml)
6. Create namespaces: kubectl create namespace minio-operator, kubectl create namespace minio-lab
7. Create secrets to private regsitry: kubectl create secret docker-registry registrycredentials --docker-server=https://registry.home.lab --docker-username=registry --docker-password= --docker-email=[email protected] -n minio-operator, kubectl create secret docker-registry registrycredentials --docker-server=https://registry.home.lab --docker-username=registry --docker-password= --docker-email=[email protected] -n minio-lab
8. Install minio operator with helm chart (helm install --namespace minio-operator operator minio-operator/operator --values minio-operator-values.yaml)
9. Install minio tenant with helm chart (helm install --namespace minio-lab minio-lab minio-operator/tenant --values minio-lab-tenant-values.yaml)

Context

High risk of usage images directly from public registry
Much longer deployment times

Regression

No idea

Your Environment

minio-operator/minio-operator 4.3.7 v4.3.7 A Helm chart for MinIO Operator
k3s: (1 control plane, 3 agents)
Client Version: v1.31.4+k3s1
Kustomize Version: v5.4.2
Server Version: v1.31.4+k3s1
cilium CNI: cilium image (running): 1.16.5
on-premis servers, Xen (xcp-ng 8.3)
Linux k3s-node-01 6.1.0-28-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.119-1 (2024-11-22) x86_64 GNU/Linux
Deployment file: https://github.com/minio/operator/blob/master/helm/tenant/values.yaml

[ramondeklein]

The error message clearly states that it's trying to pull the image from your private repository:

Failed to pull image "registry.home.lab/registry/minio:RELEASE.2024-11-07T00-52-20Z":
  failed to pull and unpack image "registry.home.lab/registry/minio:RELEASE.2024-11-07T00-52-20Z":
  failed to resolve reference "registry.home.lab/registry/minio:RELEASE.2024-11-07T00-52-20Z":
  pull access denied, repository does not exist or may require authorization:
  authorization failed: no basic auth credentials

It looks like Kubernetes is trying to pull the image from the correct repository, but it either doesn't exist or you didn't supply the correct image pull secret to access it.

[ramondeklein]

Please note that image pull secrets should be specified using the following syntax:

- name: <secret-name>

Image pull secrets are described here. You may want to post your minio-lab-tenant-values.yaml so we can check.

[bono5150]

Please note that image pull secrets should be specified using the following syntax:

• name:

Image pull secrets are described here. You may want to post your minio-lab-tenant-values.yaml so we can check.

Thank you for your response.

You can check in file tenant values file in line 39 there is no imagePullSecrets: [] in values.yaml file so the link is irrelevant here. It refers to a vector [] not a scalar {}.
The notation proposed:
- name: <secret-name>
is related to a vector and it is working perfectly fine for another values.yaml file for minio-operator operator values file line 82.
Applying this notation to tenant values.yaml we get the error:

Error: INSTALLATION FAILED: template: tenant/templates/tenant.yaml:24:9: executing "tenant/templates/tenant.yaml" at <dig "imagePullSecret" "name" "" .>: error calling dig: interface conversion: interface {} is []interface {}, not map[string]interface {}

For minio tenant values.yaml file in line 39 we have imagePullSecret: {} and this is what I am looking for.
It does not resolve the Secret type kubernetes.io/dockerconfigjson e.g: 'imagePullSecret: <secret-name>'.
It does not accept the well-known notation: - name: <secret-name>.

To be honest it is misterious for me what it accepts.

The question is: how to supply the correct image pull secret and what type of secret it should be ?

[ramondeklein]

Ah, you're right. As you can see here in the chart it should be specified like a scalar with just the name field set, so it should be something like this:

tenant:
  imagePullSecret:
    name: <secret-name-here>

Sorry for the confusion. It would have been more logical if the Tenant CRD would accept an array of pull secrets like other resources.

PS: Not it is called imagePullSecret (singular) and not imagePullSecrets (plural) like other resources.