Applying fails with: "The object has been modified; please apply your changes to the latest version and try again"

[ankon]

[2019-02-13T14:36:20.237] [INFO] kubernetes-bootstrap - extensions/v1beta1.Deployment polaris-s3-proxy-project-membership-service: smart failed: Operation cannot be fulfilled on deployments.extensions "polaris-s3-proxy-project-membership-service": the object has been modified; please apply your changes to the latest version and try again
[2019-02-13T14:36:20.238] [ERROR] kubernetes-bootstrap - Cannot apply resources: Cannot apply resource extensions/v1beta1.Deployment polaris-s3-proxy-project-membership-service through 'smart' Error: Cannot apply resource extensions/v1beta1.Deployment polaris-s3-proxy-project-membership-service through 'smart'
    at applyResource (/home/travis/build/Collaborne/collaborne/node_modules/kubernetes-bootstrap/src/bootstrap.js:435:9)
    at processTicksAndRejections (internal/process/next_tick.js:81:5)

[ankon]

It's not immediately obvious what exactly would have done that change here, although there is of course a chance that another service (such as configmapcontroller/reloaded) has touched the service due to earlier updates.

[ankon]

This happens quite consistently when indeed reloader is also touching things. We might be able to "retry" here after a while, but would have to retry from the beginning, i.e. from the original change we wanted to apply.

[ankon]

Current work-around: We restart reloader every week.

[ankon]

Idea, but only partial: It seems newer Kubernetes versions allow us to "pause"/"resume" updates on deployments (and likely daemonsets/statefulsets), so the kubernetes-bootstrap tool could do a first run over all resources and find all these, pause them, then run the updates, and then resume at the end.

This would still produce problems with the reloader though, which at the same time also wants to do updates. Pausing/resuming would hide the double restarts to the outside, but not prevent the conflicts.

[ankon]

The "pause"/"resume" will likely not work, as the changes would not get deployed -- but they would still change the resource version, which leads to the conflicts.