Trying to replace awscurl (works) with aws-curl (not yet working)

[linuxtrader]

with aws-curl apparently I need to add --service exectute-api, but it still fails...
Ran these back to back...

Success:
$ awscurl --region us-east-1 --location --request POST --header 'host: ao9ypslxuh.execute-api.us-east-1.amazonaws.com' --header 'Content-Type: application/json' --header 'Accept: /' --header 'Accept-Encoding: gzip, deflate, br' --header 'Connection: keep-alive' --data '{
"cluster_name": "dev3-compute-dev-us-east-1",
"ingress_weight": "255",
"k8s_name": "dev3-100-fltsuite-svc-data-fltsuite-svc-data",
"k8s_namespace": "dev3-fltsuite-svc-data"
}' "https://vpce-0e417798a7d74506c-brkzcqhg.execute-api.us-east-1.vpce.amazonaws.com/dev-sre/sre/ops/eks/modify-ingress-dns-weight"
Ingress dns record weight updated successfully

$ aws-curl --service execute-api --region us-east-1 --location --request POST --header 'host: ao9ypslxuh.execute-api.us-east-1.amazonaws.com' --header 'Content-Type: application/json' --header 'Accept: /' --header 'Accept-Encoding: gzip, deflate, br' --header 'Connection: keep-alive' --data '{
"cluster_name": "dev3-compute-dev-us-east-1",
"ingress_weight": "255",
"k8s_name": "dev3-100-fltsuite-svc-data-fltsuite-svc-data",
"k8s_namespace": "dev3-fltsuite-svc-data"
}' "https://vpce-0e417798a7d74506c-brkzcqhg.execute-api.us-east-1.vpce.amazonaws.com/dev-sre/sre/ops/eks/modify-ingress-dns-weight"
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method....

The Canonical String for this request should have been
'POST
/dev-sre/sre/ops/eks/modify-ingress-dns-weight

accept:/
accept-encoding:gzip, deflate, br
connection:
content-type:application/json
host:ao9ypslxuh.execute-api.us-east-1.amazonaws.com
x-amz-date:20211027T205534Z
x-amz-security-token:FwoGZXIvYXdzEA4aDOy<deleted==

accept;accept-encoding;connection;content-type;host;x-amz-date;x-amz-security-token
bf175ed8be53b4bdb6c593db867e5a99295e495cbbedc340d0e0828b7df40e32'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20211027T205534Z
20211027/us-east-1/execute-api/aws4_request
04bec624523a442fe9eb358d3b6bfb562a55cad968c23788bf1ce4970c8a706f'

Then removing all the optional headers made no diff.

aws-curl --service execute-api --region us-east-1 --request POST --header 'host: ao9ypslxuh.execute-api.us-east-1.amazonaws.com' --header 'Content-Type: application/json' --data '{
"cluster_name": "dev3-compute-dev-us-east-1",
"ingress_weight": "255",
"k8s_name": "dev3-100-fltsuite-svc-data-fltsuite-svc-data",
"k8s_namespace": "dev3-fltsuite-svc-data"
}' "https://vpce-0e417798a7d74506c-brkzcqhg.execute-api.us-east-1.vpce.amazonaws.com/dev-sre/sre/ops/eks/modify-ingress-dns-weight"
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method.

[sormy]

So, you are trying to invoke API on private api gateway located in another AWS account behind VPC endpoint?
I guess the culprit is missing support for custom host name override using header. Should be fixable.

[sormy]

Please try test version from custom-host branch. If it works, i will polish code a bit and merge to master.

[linuxtrader]

Set up custom-host as shown in image attached.

#New error message...

16:35 $ aws-curldev3_us-east-1_255
{"message":"Forbidden"}✔ ~/|✔]

#And confirmed still working in status quu
16:37 $ dev3_us-east-1_255
Ingress dns record weight updated successfully

[sormy]

are you sure you have valid AWS credentials in ENV variables? aws-curl doesn't read ~/.aws/credentials.
Could you please post output of aws-curl --verbose ...?

[linuxtrader]

I will run that verbose. Yes I’m sure the credentials are in environment variables. That’s required by awscurl also.

…

-- Daniel Rose 972-342-5208 cell

On Oct 29, 2021, at 5:09 PM, Artem Butusov ***@***.***> wrote:  are you sure you have valid AWS credentials in ENV variables? aws-curl doesn't read ~/.aws/credentials. Could you please post output of aws-curl --verbose ...? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[sormy]

I might need to see also awscurl verbose response to compare the payload.

[linuxtrader]

Here are three files, the functions defined, and the verbose output of each....

function.txt
aws-curl_verbose.txt
awscurl_verbose.txt

[sormy]

I have published new revision 0eb5bf8 for custom-host branch. Please try and let me know.

[linuxtrader]

No improvement, output attached...
output.txt

[sormy]

Try without connection header.

[linuxtrader]

Now it says...
{"message":"Credential should be scoped to correct service: 'execute-api'. "}

[sormy]

Try to pass explicitly “—service execute-api”

[linuxtrader]

Hooray it works now!

I had --service set explicitly before, and must have deleted that part while removing the connection header.

[sormy]

Thanks for confirmation. I will keep the issue open until I will merge the change to the main branch.

[linuxtrader]

By the way, 2 minor differences remains..., which don't bother me.

1. awscurl, first, provides a linefeed, where aws-curl 2nd, does not...

✔ ~/Documents/aws
21:26 $ ./dev3westOFF
Ingress dns record weight updated successfully
✔ ~/Documents/aws
21:26 $ vi dev3westOFF
✔ ~/Documents/aws/
21:27 $ ./dev3westOFF
Ingress dns record weight updated successfully✔ ~/Documents/aws/

2. For aws-curl to work I had to remove both the connection header and the Accept-Encoding header.
   I slowly added headers back and it broke on Accept-Encoding.

Just FYI as I can live without those headers.

[sormy]

1. Did not get what you mean. aws-curl sends requests as it is.

2. It is pretty natural to not pass explicitly Accept-Encoding and Connection headers since it is something that curl is responsible for. You can't safely pass unsupported by vanilla curl Accept-Encoding but curl has an option to send Accept-Encoding: gzip and can decompress response with --compressed flag. Using custom Accept-Encoding will likely require to manually pipe curl output to another process that can decode compressed output. Also keep alive doesn't make any sense for just one curl call. However vanilla curl can execute multiple requests using the same TCP connection but it is not implemented in aws-curl yet.