sync on-demand: docker pull fails for OCI image index(application/vnd.oci.image.index.v1+json)
#2836
Labels
Comments
As I asked in #2811, what call is docker actually making to the zot server? Can you share more of the logs? |
andaaron
changed the title
docker pull fails for OCI image index(application/vnd.oci.image.index.v1+json)
andaaron
changed the title
docker pull fails for OCI image index(application/vnd.oci.image.index.v1+json)docker pull fails for OCI image index(application/vnd.oci.image.index.v1+json)
|
we have an ECR registry and have configured Zot as a pull-through cache (on-demand). However, when attempting a docker pull through Zot for the following OCI manifest, the pull fails: While docker pull directly from ECR works, the same operation through Zot fails. I do not believe this is a client-side issue. Below are relevant log messages for reference
This log line indicates that /v2/appname/manifests/1590357999 was not in the accepted media types. However, the image manifest (as confirmed by the manifest inspect I shared earlier) shows that it is in one of the accepted media types. looks like an bug. still I didn't had a chance to review the code. |
|
Question is why it fails to get the image details. It is possible it fails to get the manifest from ecr (instead of having an issue with the manifest content)? Maybe it errors here https://github.com/project-zot/zot/blob/main/pkg/extensions/sync/remote.go#L132 ? |
|
I have verified the following:
|
|
The strange part is it fails before our checks for manifest media type value. |
|
@andaaron Thanks for linking the relevant code path. I will run a few more tests tomorrow and provide an update here. I appreciate your quick response—it’s really helpful! |
|
Some other limitation about the way sync works with multiarch at the moment:
There's a PR for replacing the libraries used for sync, but due to limited time on the part of the contributor, it has been dragging for a long time. |
|
Could you please share the PR link? I’ll give it a try with that branch |
|
This is the PR, but I don't think it's ready to be used at the moment: #2524 |
|
Hi @tamilhce, I have tried to reproduce this issue, with a more basic configuration (as I don't have access to ECR, and also the zot configuration was not provided in the bug description). I tried using 2 zot instances. 1st zot has the config: 2nd zot has the config: I uploaded images to the 1st zot using: I checked the repo/image/tag is present on zot running on port 5000: After that I pulled from the 2nd zot. |
|
I also tried using a build from your branch |
|
Hi @andaaron, |
|
Hi @andaaron, sure I will reprod the issue locally and share the steps to proceed further. |
|
Note we want to merge #2903 soon. Which is a substantial rewrite of the sync logic. |
|
I tried reproducing the issue locally. This is what I observed: When I copied the image from ECR to Zot1 and used Zot as the upstream registry, configuring Zot2 for sync on-demand exactly as you mentioned, it worked without any issues for the same image. zot version zot1 zot2
aws ecr get-authorization-token | jq -r '.authorizationData[0].authorizationToken' | base64 -d | cut -d: -f2 | skopeo login -u AWS --password-stdin accountid.dkr.ecr.us-east-1.amazonaws.com skopeo --insecure-policy copy --dest-tls-verify=false --format=oci --all docker://test-image:1542762016 docker:// zot-reg.net:5000/test-image:1 on-demand sync it works fine as expected.
zot configuration for the ECR as upstream Trying out on-demand sync with ECR as upstream registry
Relevant zot error logs still not sure, why zot fails for the OCI image index(application/vnd.oci.image.index.v1+json) only when ECR is configured as upstream. |
|
Any chance this does not reproduce with #2903? |
|
I haven't tried with #2903 yet. I'll check and update this thread. |
|
with #2903, initially I observed the following error Once I added the required IAM policy in the repo, the sync works as expected. logs However, even though the images is cached in the s3. Each time it fetch's sync from the ECR, & observed following details in the logs relevant logs Since for each on-demand request, Zot thinks that the "remote image digest changed" and pulls the image again, I observed that it takes more time to pull the image compared to directly accessing the ECR. |
|
Is there something different inside the 2 manifests? Maybe the order of properties in the json dictionaries? Only docker media types would be rewritten as oci media types, but the reported issue is for an oci media type. Since there as an update to the IAM policy, maybe the main branch implementation sync works as expected? |
with docker pull zot.net/test:1542762016
I will try to debug further and get more details. |
I think while checking the image mainfest, we are not passing the tag for the upstream registry. we are passing the tag in the image store. Since the tag is not being passed, it might default to the latest tag, causing a mismatch between the local digest and the upstream registry. |
|
@petu^^
…On Wed, Feb 5, 2025, 17:10 tamil vanan ***@***.***> wrote:
func (registry *DestinationRegistry) CanSkipImage(repo, tag string, digest
godigest.Digest) (bool, error) {
// check image already synced
imageStore := registry.storeController.GetImageStore(repo)
_, localImageManifestDigest, _, err := imageStore.GetImageManifest(repo,
tag)
https://github.com/eusebiu-constantin-petu-dbk/zot/blob/sync_regclient_2/pkg/extensions/sync/destination.go#L58
I think while checking the image mainfest, we are not passing the tag for
the upstream registry. we are passing the tag in the image store. Since the
tag is not being passed, it might default to the latest tag, causing a
mismatch between the local digest and the upstream registry.
—
Reply to this email directly, view it on GitHub
<#2836 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGEG7UHGDKYCB426UGN66P32OISXTAVCNFSM6AAAAABTYSVJ6OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMZXGE2DIOBXGA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@andaaron/ @eusebiu-constantin-petu-dbk , you can ignore this comment @andaaron , However, in the ECR image index, the digest is: This indicates that the digest has not been calculated correctly for the locally stored image. |
I can't check the content of the files since I don't have access to that image. Can you please check what is different in the remote index vs the local index content? L.E. what is the input over which the hash "sha256:01163f3d0e81b5eb2e9ecbffc1a2534db4527a06008ef7bfe13085936285d3f8" was generated. versus the one for "sha256:4bc9139c512e40d87d25f80830627713a83db1d12f3df6fc8f0cece5dd580e2a". |
|
I tried pulling the image from both the ECR and the Zot registry to the local disk and performed a diff.
Files content/blob looks same I found following diff for the manfiest between the ecr and the zot Manifest digest |
|
Question is why is that a "application/vnd.docker.distribution.manifest.list.v2+json? We convert this media type to and OCI image index. |
|
There is "docker image index" containing "OCI image manifests" in ecr vs all OCI in zot (as expected)? |
|
In our build pipeline, we use the manifest-tool to create a multi-architecture image, which is then pushed to ECR. |
|
I used the following command to create an OCI image index and pushed it to ECR. Even though I explicitly specified the OCI format, the resulting manifest in ECR is stored as a Docker manifest list (application/vnd.docker.distribution.manifest.list.v2+json). Pushed the manifest using: However, inspecting the manifest in ECR reveals that AWS ECR stores it as a Docker manifest list instead of an OCI image index: https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-manifest-formats.html |
|
Hmm, can you try Log term we were discussing and this is what we would want to do: #2903 (comment) (probably in a later PR). |
Thanks, @andaaron! It works now. 😊 I can now proceed further and update my pipeline to create new images using the OCI image index. However, we have a large number of existing repositories and images that need updating, which isn't feasible. I’d like to work on this feature. yo can assign it to me. |
|
I think @eusebiu-constantin-petu-dbk already has some code for it, but it depends on his existing pr. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Discussed in #2811
Originally posted by tamilhce November 28, 2024
I was running zot with following patch version https://github.com/tamilhce/zot/pull/13/ as a pull through for ECR
I have verified that some multi-arch images are working as expected, while others are not. Upon further investigation, I found that container multi-arch images with the type
application/vnd.oci.image.index.v1+json(the OCI manifest list for multi-platform images) are not supported. Only multi-arch images with the Docker manifest format,application/vnd.docker.distribution.manifest.list.v2+json(the Docker manifest list for multi-platform images), are supported.AWS ECR supports both the OCI and Docker manifest formats, whereas Zot only supports the multi-arch/image index of the Docker manifest format.
working -
application/vnd.docker.distribution.manifest.list.v2+jsonNot working -
application/vnd.oci.image.index.v1+jsonis this the expected behavior? If it's a bug, I will work on the feature. Looking forward to your valuable reply.
The text was updated successfully, but these errors were encountered: