Update Operator & Tenant at the same time fails

[tmohme]

I try to update mini-operator from 6.0.3 to 7.0.0 and mini-tenant from RELEASE.2024-08-26T15-33-07Z to the current RELEASE.2025-01-20T14-49-07Z.
The environment is K8s with ArgoCD deploying Helm-Charts.

This is more or less the same as #2305.

Expected Behavior

I expect the mini-operator to update to 7.0.0 and the operator to pick-up the change in the Tenant CR from "spec.image=quay.io/minio/minio:RELEASE.2024-08-26T15-33-07Z" to "spec.image=quay.io/minio/minio:RELEASE.2025-01-20T14-49-07Z", subsequently change the Tenant-Pool-StatefulSet and finally restarting its pods so that they run with the new image.

Current Behavior

The mini-operator successfully updates to 7.0.0 and the Tenant CR is also changed to "spec.image=quay.io/minio/minio:RELEASE.2025-01-20T14-49-07Z" as expected.
But then the operate seems to send an "update" to the minio servers, which fail to execute it due to missing access rights/wrong image layout:
server update failed with: open /usr/bin/.minio.check-perm: permission denied, do not restart the servers yet.

Possible Solution

A workaround is, to simply kill the STS and let the operator re-create it, then with the correct minio-image.
A solution would be, to let the operator patch the STS and restart the pods.

When I keep the operator version 6.0.3 and only update the minio image, the operator continuously logs an error message
main-controller.go:1570] error synching 'minio/minio-tenant': host mini-tenant-pool01-1.minio-tenant-hl.minio.svc.cluster.local:9000: Server is running the latest version: 2025-01-20T14:49:07Z;... and the status.currentStatus of the Tenant CR alternating between Updating MinIO Version and host mini-tenant-pool01-1.minio-tenant-hl.minio.svc.cluster.local:9000: Server is running the latest version: 2025-01-20T14:49:07Z;..., never getting "ready".
Therefore, this is not a solution for us.

Steps to Reproduce (for bugs)

1. Deploy minio-operator 6.0.3
2. Deploy minio:RELEASE.2024-08-26T15-33-07Z
3. Update the operate Helm chart to 7.0.0 and minio to RELEASE.2025-01-20T14-49-07Z

Context

Consequently using GitOps and infrastructure-as-code we don't want to modify running images at all.
Instead we want to configure a new version of the app and replace the old pod with the new ones to ensure full reproducibility.

Regression

don't know

Your Environment

• Version used (minio-operator): 6.0.3 => 7.0.0
• Environment name and version (e.g. kubernetes v1.17.2): v1.31.0
• Server type and version:
• Operating System and version (uname -a): Linux mini-tenant-pool01-1 6.12.9-talos Add Minio operator  #1 SMP x86_64 GNU/Linux
• Link to your deployment file:

[ramondeklein]

We have had some MinIO versions that had problems with directory permissions and couldn't be updated this way. They need a restart of the statefulset. Newer MinIO versions don't have this issue anymore, so it should be fine once you upgrade to a newer MinIO version...

This is not an operator issue, but was an issue in the MinIO image itself.

[ramondeklein]

Unless the image is specified in the Tenant resource, the operator will use the latest MinIO image that was available at time of the operator release. If you want to fix the MinIO image, then set the .spec.image value in the Tenant resource. Make sure you keep your MinIO up-to-date though...

[tmohme]

This is not a problem of the broken minio:RELEASE.2024-08-26T15-33-07Z image.

Meanwhile I tested this combined update in three different cloud environments with stable results.
In two of the environments, the operator changes the .spec.template.spec.containers[name=minio].image value of the tenant's stateful set (and everything works as expected) and in one environment this does not happen.
All installations are set up as identical as possible, from the identical specification in a git repo without manual intervention.

The .spec.image value in the Tenant resource has the correct value in all three environments.
It is just that, in one environment, the operator seemingly does not pick up and use this value to update the .spec.template.spec.containers[name=minio].image value of the tenant's stateful set.
Maybe this is caused by some kind of race condition due to different timing as the failing environment is the fastest.
Instead we have to manually patch this value in the stateful set to make it run the desired minio server image version.

For now, I understand we sometimes have to use this manual workaround of patching the stateful set to actually update the tenant.

[ramondeklein]

The problem that MinIO couldn't write in /usr/bin was patched with minio/minio@229f04a.