Cannot Deploy Endpoint

[goldmermaid]

Describe the bug

Error hosting endpoint xxxxxx: Failed. Reason:  Please make sure all images included in the model for the production variant AllTraffic exist, and that the execution role used to create the model has permissions to access them..

To reproduce

sagemaker_model = MXNetModel(model_data=trained_model_upload, 
                             image='emnlp:opt', # docker images
                             role=sagemaker.get_execution_role(), 
                             py_version='py3',            # python version
                             entry_point='serve.py',
#                              source_dir='.'
                            )

Please see more details here: https://github.com/goldmermaid/KDD2020/blob/master/EMNLP/train_deploy_bert.ipynb.

Expected behavior
Deployed to endpoint.

System information
Sagemaker 2.3.0

• SageMaker Python SDK version:
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): MXNet
• Framework version: 1.6.0
• Python version: 3.6
• CPU or GPU: GPU
• Custom Docker image (Y/N): Y

Additional context
Thanks for the help!

[metrizable]

Hello @goldmermaid

Thank you for using Amazon SageMaker. I see that you have specified image emnlp:opt. The error message indicates there may be an issue with it:

Please make sure all images included in the model for the production variant AllTraffic exist, and that the execution role used to create the model has permissions to access them.

Can you confirm 1) the existence of the image uri you have specified, and 2) that the execution role you are using has access to it?

Best regards

[goldmermaid]

Hi @metrizable , thank you for the quick reply. I confirm the image uri exists. (I checked by calling docker image ls, and this image is there.) For your second question, I am not exactly sure whether I set access correctly. Could you give me some hints on how to grant access? Thanks!

[metrizable]

@goldmermaid

Just to confirm, are you specifying the full URI (unique resource identifier) of the image (should be something like <aws_account_id>.dkr.ecr.<your-region>.amazonaws.com/<image-name>:<tag>)? From here, it doesn't look like it. This example notebook uses a custom image specifying the full image URI.

[lu-liu-rft]

Hi I got the same error when using

from sagemaker.tensorflow.serving import Model

model = Model(model_data=model_data,
              role=role,
              framework_version='1.15.2',
              sagemaker_session=sagemaker_session,
              name=name)

predictor = model.deploy(initial_instance_count=1,
    instance_type='xxx',
    endpoint_name=name,
    update_endpoint=False)

I did add the following policies to my SageMaker execution role:

{
            "Effect": "Allow",
            "Action": [
                "ecr:SetRepositoryPolicy",
                "ecr:CompleteLayerUpload",
                "ecr:BatchGetImage",                
                "ecr:BatchDeleteImage",
                "ecr:UploadLayerPart",
                "ecr:DeleteRepositoryPolicy",
                "ecr:InitiateLayerUpload",
                "ecr:DeleteRepository",
                "ecr:PutImage"
            ],
            "Resource": "arn:aws:ecr:*:*:repository/*sagemaker*"
        },

When I still have this error and couldn't deploy the model?

[wissambenhaddad]

Hello, any updates on this ?

[lu-liu-rft]

I resolved it but I can't remember what I did differently to be honest (it's been a bit long now...), but I think you can try to remove the sagemaker in the resources, so you can try to use "Resource": "arn:aws:ecr:*:*:repository/*" in your script, it will tell the SageMaker execution role to access to any repositories.

[wissambenhaddad]

I fixed it just now, somehow, after I trained my custom model and deployed and endpoint using a custom inference image that I modified later, the image was not recognized because I deleted the old one but kept the same name and tag. Maybe sagemaker checks the sha1 or something.