Object Store: Anonymous Access to S3

[tasansal]

Hi everyone,

I am trying to have a generic interface to read from gs/s3/az/http links using ObjectStore::get_ranges()

I construct the dynamic ObjectStore like this:

let uri = "gs://bucket/prefix" ;

let url = Url::parse(uri)?;
let (store, path) = object_store::parse_url(&url)?;

This seems to work great for GCS and HTTP.

However, when I pass a public S3 (HTTP) link like this:
http://s3.amazonaws.com/teapot/filt_mig.sgy

let uri = "http://s3.amazonaws.com/teapot/filt_mig.sgy" ;

let url = Url::parse(uri)?;
let (store, path) = object_store::parse_url(&url)?;

I am getting this error.

Error: Generic HTTP error: Request error: Error after 0 retries in 6.208µs, max_retries:10, retry_timeout:180s, source:builder error for url (http://s3.amazonaws.com/teapot/filt_mig.sgy)

Caused by:
    0: Request error: Error after 0 retries in 6.208µs, max_retries:10, retry_timeout:180s, source:builder error for url (http://s3.amazonaws.com/teapot/filt_mig.sgy)
    1: Error after 0 retries in 6.208µs, max_retries:10, retry_timeout:180s, source:builder error for url (http://s3.amazonaws.com/teapot/filt_mig.sgy)
    2: builder error for url (http://s3.amazonaws.com/teapot/filt_mig.sgy)
    3: URL scheme is not allowed

AFAIK, based on crate documentation, object_store tries to convert this to s3 link internally (there is an example for cloudflare). But not allowing it for S3?

It is very likely I am doing something wrong because I am a Rust noob, but just wanted to check with the community.

I also got the s3:// protocol link for the same path, and I get a slightly different error.

let uri = "s3://open.source.geoscience/open_data/teapot/filt_mig.sgy" ;

let url = Url::parse(uri)?;
let (store, path) = object_store::parse_url(&url)?;

Error: Generic S3 error: Error after 10 retries in 5.837870792s, max_retries:10, retry_timeout:180s, source:error sending request for url (http://169.254.169.254/latest/api/token)

Caused by:
    0: Error after 10 retries in 5.837870792s, max_retries:10, retry_timeout:180s, source:error sending request for url (http://169.254.169.254/latest/api/token)
    1: error sending request for url (http://169.254.169.254/latest/api/token)
    2: client error (Connect)
    3: tcp connect error: Host is down (os error 64)
    4: Host is down (os error 64)

I am curious why this is happening? Normally I have to set anonymous access to True when using s3fs in Python, but I see no such option in object_store. Is this the problem?

[tustvold]

https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html#method.with_skip_signature

Will resolve the second issue, as for the URL parsing error perhaps try with https

[tasansal]

@tustvold, thanks! the skip_signature option works. I was able to generalize it to work with generic interface with parse_url_opts.

let (store, path) = parse_url_opts(&url, [("skip_signature", "true")])?;

However, the HTTP link still doesn't work. Both HTTP and HTTPS give the following errors:

HTTP

Error: Generic HTTP error: Request error: Error after 0 retries in 5.333µs, max_retries:10, retry_timeout:180s, source:builder error for url (http://s3.amazonaws.com/teapot/npr3_field.sgy)

Caused by:
    0: Request error: Error after 0 retries in 5.333µs, max_retries:10, retry_timeout:180s, source:builder error for url (http://s3.amazonaws.com/teapot/npr3_field.sgy)
    1: Error after 0 retries in 5.333µs, max_retries:10, retry_timeout:180s, source:builder error for url (http://s3.amazonaws.com/teapot/npr3_field.sgy)
    2: builder error for url (http://s3.amazonaws.com/teapot/npr3_field.sgy)
    3: URL scheme is not allowed

HTTPS

Error: Generic S3 error: URL did not match any known pattern for scheme: https://s3.amazonaws.com/teapot/npr3_field.sgy

Caused by:
    URL did not match any known pattern for scheme: https://s3.amazonaws.com/teapot/npr3_field.sgy

Any ideas?

[tustvold]

It is missing the region, the supported URL formats can be found - https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html#method.with_url