Failed to push image: error copying layers and metadata from "containers-storage[...]

[eduardolucioac]

Hi all!

I'm trying to create an application with the commands...

oc new-project spring-boot-sample
oc new-app codecentric/springboot-maven3-centos~https://github.com/codecentric/springboot-sample-app.git

...but I am constantly getting this error...

Successfully pushed image-registry.openshift-image-registry.svc:5000/spring-boot-sample/springboot-sample-app:latest
Warning: Push failed, retrying in 5s ...
Getting image source signatures
Copying blob sha256:5c0423d6ef09c20b871d3490c812fa8cb5482e8e06656d212b20c6d0daa6588a
Copying blob sha256:dca33d095b85b196af223992b93177adb721a78a0100542720cff4ead55d6d4a
Copying blob sha256:a2725f86b366b2b0a5ef0de3ae8f14c052a654bd1fc968ef4fff2ecfd02c3b82
Successfully pushed image-registry.openshift-image-registry.svc:5000/spring-boot-sample/springboot-sample-app:latest
Warning: Push failed, retrying in 5s ...
Getting image source signatures
Copying blob sha256:ac87fd5d970d3b9557577ab5b94ce2db461d03bfccd8011a6565ad19571e7927
Copying blob sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729
Copying blob sha256:556bcfa9b24935162506e999b96e937c26c761caae955d45e9150bb7e443f044
Copying blob sha256:dca33d095b85b196af223992b93177adb721a78a0100542720cff4ead55d6d4a
Copying blob sha256:734fb161cf896cf5c25a9a857a4b4d267bb5a59d5acf9ba846278ab3f3d1f5d5
Successfully pushed image-registry.openshift-image-registry.svc:5000/spring-boot-sample/springboot-sample-app:latest
Warning: Push failed, retrying in 5s ...
Registry server Address: 
Registry server User Name: serviceaccount
Registry server Email: [email protected]
Registry server Password: <<non-empty>>
error: build error: Failed to push image: error copying layers and metadata from "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared]image-registry.openshift-image-registry.svc:5000/spring-boot-sample/springboot-sample-app:latest" to "docker://image-registry.openshift-image-registry.svc:5000/spring-boot-sample/springboot-sample-app:latest": Error writing blob: failed to read from destination repository spring-boot-sample/springboot-sample-app: 400 (Bad Request)

NOTE: I have been testing for other applications but the same issue occurs.

QUESTION: Does anyone know what might be going on?

[]'s

[eduardolucioac]

Same problem, but now trying another project...

oc new-project wordpress-test
oc new-app centos/php-73-centos7~https://github.com/WordPress/WordPress.git

Pushing image image-registry.openshift-image-registry.svc:5000/wordpress-test/wordpress:latest ...
Getting image source signatures
Successfully pushed image-registry.openshift-image-registry.svc:5000/wordpress-test/wordpress:latest
Warning: Push failed, retrying in 5s ...
Getting image source signatures
Successfully pushed image-registry.openshift-image-registry.svc:5000/wordpress-test/wordpress:latest
Warning: Push failed, retrying in 5s ...
Getting image source signatures
Copying blob sha256:af5ca36785f540d0ca92e3ae4d56ee3a63849a17beba5e2b8ace98646b38cd2d
Copying blob sha256:9dbbd3eec73052e9c971f3094e3f00badfef563367e85b443f85f363e7575ddb
Copying blob sha256:9e50ed62ad7d3c60dfbd04d594597f788757c1f35a8dba1d874282d5e5af51dc
Successfully pushed image-registry.openshift-image-registry.svc:5000/wordpress-test/wordpress:latest
Warning: Push failed, retrying in 5s ...
Registry server Address:  
Registry server User Name: serviceaccount
Registry server Email: [email protected]
Registry server Password: <<non-empty>>
error: build error: Failed to push image: error copying layers and metadata from "containers-storage:[overlay@/v
ar/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared]image-registry.openshif
t-image-registry.svc:5000/wordpress-test/wordpress:latest" to "docker://image-registry.openshift-image-registry.
svc:5000/wordpress-test/wordpress:latest": Error trying to reuse blob sha256:b2cc5146c9c7855cb298ca8b77ecb153d37
e3e5c69916ef423613a46a70c0503 at destination: unable to retrieve auth token: invalid username/password: unauthor
ized: unable to validate token: Forbidden

[]'s

[eduardolucioac]

I have also observed that this error has happened constantly...

P cloud-credential-operator-5f794bc9c-bmzbb
NS openshift-cloud-credential-operator
CrashLoop BackOff

[eduardolucioac]

More information about the possible problem...

[root@okd-services edulucio]# oc logs -n openshift-cloud-credential-operator -f cloud-credential-operator-5f794bc9c-bmzbb -c kube-rbac-proxy
[...]
2021/08/18 19:21:56 http: proxy error: dial tcp 127.0.0.1:2112: connect: connection refused
2021/08/18 19:22:03 http: proxy error: dial tcp 127.0.0.1:2112: connect: connection refused
2021/08/18 19:22:26 http: proxy error: dial tcp 127.0.0.1:2112: connect: connection refused
2021/08/18 19:22:33 http: proxy error: dial tcp 127.0.0.1:2112: connect: connection refused
2021/08/18 19:22:56 http: proxy error: dial tcp 127.0.0.1:2112: connect: connection refused
2021/08/18 19:23:03 http: proxy error: dial tcp 127.0.0.1:2112: connect: connection refused

[root@okd-services edulucio]# oc logs -n openshift-cloud-credential-operator -f cloud-credential-operator-5f794bc9c-bmzbb -c cloud-credential-operator
Copying system trust bundle
/bin/bash: line 4: /usr/bin/cloud-credential-operator: Success

[]'s