Pushing images doesn't seem to be supported.

[isaacegglestone]

Pushing images doesn't seem to be supported.

After adjusting the IAM role to allow for writes to the repo I'm still getting:

error parsing HTTP 403 response body: no error details found in HTTP response body: "{"message":"Missing Authentication Token"}"

Acquiring a login via IAM doesn't seem to help:
$(aws ecr get-login --no-include-email --region us-west-2)

I thought this might have been because the above using the normal AWS formed domain name however separating it as such:

aws ecr get-login --no-include-email --region us-west-2

Then adjusting the domain output to be the correct one for running it as such:

Original:
docker login -u AWS -p <the_password_it_provides> https://<your_aws_account>.dkr.ecr.us-west-2.amazonaws.com

Changed to:
docker login -u AWS -p <the_password_it_provides> https://docker.mydomain.io

The above login succeeds, however when actually performing the push:
It throws the following error:
error parsing HTTP 403 response body: no error details found in HTTP response body: "{"message":"Missing Authentication Token"}"

Notice it doesn't give the regular docker login failed message, so this seems like the lambda process is not really sending the auth token from the client.

Looking at the lambda.js file seems to confirm no handling of PutImage in the code.

Expectations:
This is a great idea and super useful.
To be a fully functional front for a docker repo though we probably need to pass the auth token and add PutImage among some other capabilities. Otherwise not fully useful.

Extras:
Ideally we have a duplicated in terraform for a more wider audience.

Proposal: I'll see if I can add the required js code for the putimage and other write features.
We can have a paramater or something to exclude these if someone really just wants a read only repo.

[monken]

Correct, the current version doesn't support push. You can still push to the underlying ECR repository. API Gateway supports pass-through IAM credentials so your approach should work.

I like the proposal. Please keep in mind that we should keep the lambda function code at or below 4096bytes to be able to inline the code into the template. A second lambda function for putImage might be necessary. Otherwise, we'd have to distribute the code on an S3 bucket which is a pain as it needs to be replicated across all regions.

[monken]

regarding terraform, I'd prefer a simple terraform wrapper template that makes use of the CloudFormation provider (https://www.terraform.io/docs/providers/aws/r/cloudformation_stack.html). I'd rather maintain everything in one place instead of trying to keep two deployment methods in sync.

[monken]

I looked into the docker spec for pushing images (https://docs.docker.com/registry/spec/api/#pushing-an-image) and the ECR API (https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/ECR.html#uploadLayerPart-property) and it seems like we have to proxy the actual image blob through the API Gateway (for pulling images, we only forward the S3 object location which is much more efficient). This is not ideal for many reasons:

• it's slow because we add an additional hop
• the API Gateway has a hard 10MB limit for payloads, so we need to implement the fairly complex chunked upload spec
• additional cost apply because the lambda function runs for a longer period of time to stream the data to the ECR api

[isaacegglestone]

Thanks for the full and proper review of the issue , agreed not straightforward. Internally I am probably going use another approach which implements nginx as aproxy to the AWS ECR, LB and instances :( as there is already internal modules for that and I can't justify the coding time of the above sadly. As much as I would want to do that.

[isaacegglestone]

Have you thought of removing lambda from the picture and just using the api gateway http proxy method? https://aws.amazon.com/blogs/aws/api-gateway-update-new-features-simplify-api-development/

[isaacegglestone]

Okay so it turns out AWS has a direct API gateway to ECR integration feature. The lambda code you have developed is great, but requires a lot of work to get the full features. However When removing all methods and replacing it with this with ANY and pointed at the ECR integration I'm still faced with this token "Missing Authentication Token".

Upside is I can now raise a support case on this to figure out why this doesn't work.

[monken]

Interesting approach! Let me know what you find out!

[isaacegglestone]

Still working with AWS on this since 11 days ago but just updating to let you know I haven't dropped this. So far they have not been able to provide working example with this integrtion. Waiting to hear back from ECR team on this.

[isaacegglestone]

Just an update on this. I finally got an example solution from them that will allow me to use authentication to ECR. I have not tested it to see if it works but a quick glance over looks like it will. I finally got some decent feedback from them after a lot of back and foreth and several AWS support people trying to get an example working for me, I think they finally have something. So I will test their basic solution out and then try to raise a PR with the cloudformation version if it looks possible.

[monken]

That's great news! Keep me posted!

[ryderjgillen]

@isaacegglestone - any update on this I too have been looking into this and want to determine if this is viable approach.

[fred-vogt]

What was the outcome @isaacegglestone ?

[isaacegglestone]

@isaacegglestone - any update on this I too have been looking into this and want to determine if this is viable approach.

Sorry for the extremely late reply.

In the end they told me it was not possible after I tried what they suggested and it didn't work.

They told me it simply wasn't supported to run it with a custom domain name because the token itself includes some kind of checksum on the ECR domain name or something within their backend was blocking the auth without the correct domain name.

We ended up deploying harbor in kubernetes backed on s3 and with caching to make it faster.
Basically gave up on ECR.

Keep in mind this was some time ago now and things change within AWS fast so they may have resolved this issue.

[isaacegglestone]

@isaacegglestone - any update on this I too have been looking into this and want to determine if this is viable approach.

Sorry for the extremely late reply.

In the end they told me it was not possible after I tried what they suggested and it didn't work.

They told me it simply wasn't supported to run it with a custom domain name because the token itself includes some kind of checksum on the ECR domain name or something within their backend was blocking the auth without the correct domain name.

We ended up deploying harbor in kubernetes backed on s3 and with caching to make it faster.
Basically gave up on ECR.

Keep in mind this was some time ago now and things change within AWS fast so they may have resolved this issue.

Ah seems not: aws/containers-roadmap#299
And your the last one to post on this issue lol
Maybe I ran into a credential helper issue but I don't remember ever using it.