Unable to benchmark custom s3 endpoint from openstack vm

[bojleros]

Hi,

Environment setup:
testvm => 2x harpoxy-s3 => 3x ceph radosgw

Rados gw runs on dedicated hardware while testvm and haproxy-s3 runs on openstack. This setup works good since i was able to use rclone to POST/GET objects on this custom s3. Note that radosgw are completly standalone and not integrated with openstack keystone at all.

Problem definition:
As a second step i wanted to run s3-benchmark just to check overall long term performance.

This is how my credentials are set:

[fedora@brtest ~]$ cat ~/.aws/credentials 
[brtest]
aws_access_key_id = xxx
aws_secret_access_key = xxxx

I get something like that:

[fedora@brtest ~]$ ./s3-benchmark --endpoint s3.ourdomain.io --region custom-signing-region --bucket-name benchmark

--- SETUP --------------------------------------------------------------------------------------------------------------------

panic: Filed to create S3 bucket: EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request
caused by: 404 Not Found

The resource could not be found.

   

goroutine 1 [running]:
main.setup()
	/Users/daniel/go/src/github.com/dvassallo/s3-benchmark/main.go:229 +0xa35
main.main()
	/Users/daniel/go/src/github.com/dvassallo/s3-benchmark/main.go:113 +0x44

I believe the issue is caused by fact that aws-go-sdk recognizes our openstack api as a valid aws api and tries to obtain s3 credentials this way.

fedora@brtest ~]$ curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-action
instance-id
instance-type
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
public-keys/
reservation-id

Would it possible to force s3-benchmark to skip aws autodetection ??

Regards,
Bart

[bojleros]

Btw,

This is what i get from baremetal host located outside of openstack:

~ $ ./s3-benchmark --endpoint s3.prd.ams01.mgas.io --region custom-signing-region --bucket-name brtest

--- SETUP --------------------------------------------------------------------------------------------------------------------

panic: Filed to create S3 bucket: EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

goroutine 1 [running]:
main.setup()
	/Users/daniel/go/src/github.com/dvassallo/s3-benchmark/main.go:229 +0xa35
main.main()
	/Users/daniel/go/src/github.com/dvassallo/s3-benchmark/main.go:113 +0x4

[engel75]

did you try to set the credentials in your env like:

export AWS_ACCESS_KEY_ID="HSGSGDHJTASDCDSGHR"
export AWS_SECRET_ACCESS_KEY="gdhahadshadHDhadhjDShjdhjdJdsHDSBdSAgr"

I was able to benchmark RadosGW that way.

[liberodark]

Hi,

Have the same issue with export option.

--- SETUP --------------------------------------------------------------------------------------------------------------------

panic: Filed to create S3 bucket: RequestError: send request failed
caused by: Put s3.eu-west-3.amazonaws.com/synergi2-ged: unsupported protocol scheme ""

goroutine 1 [running]:
main.setup()
	/Users/daniel/go/src/github.com/dvassallo/s3-benchmark/main.go:229 +0xa35
main.main()
	/Users/daniel/go/src/github.com/dvassallo/s3-benchmark/main.go:113 +0x44

Best Regards