'Unsupported Data Source' adding GDAL AWS S3 raster layers with credential options

[rtmorgan]

What is the bug or the crash?

I am unable to add an AWS S3 raster layer (Layer -> Add Layer -> Add Raster Layer -> Protocol Type AWS S3) with the protocol Credential Options (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) set to the appropriate values.

The following message is generated (sanitized with 'xxxxx'):

"Unsupported Data Source: /vsis3/xxxxx/xxxxx/xxxxx.tif|credential:AWS_ACCESS_KEY_ID=xxxxx|credential:AWS_SECRET_ACCESS_KEY=xxxxx is not a supported raster data source"

I am however able to add the AWS S3 raster layer if AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set as QGIS system-level environment variables (Settings -> Options -> System -> Environment).

Steps to reproduce the issue

Click on (Layer -> Add Layer -> Add Raster Layer)
Within the 'Source Type' section, select 'Protocol: HTTP(s), cloud, etc.'Protocol Type AWS S3)
Within the 'Protocol' section, select 'AWS S3' from the 'Type' drop-down
Within the 'Protocol' section, enter the bucket name in the 'Bucket or container' field
Within the 'Protocol' section, enter the object key name in the 'Object key' field
Within the 'Credential Options' section, select 'AWS_ACCESS_KEY_ID' from the drop-down and enter the AWS Access Key ID value as the attribute value
Within the 'Credential Options' section, on the second row - select 'AWS_SECRET_ACCESS_KEY' from the drop-down and enter the AWS Secret Access Key value as the attribute value
Click on the 'Add' button at the bottom of the window.

Versions

QGIS version	3.40.3-Bratislava
QGIS code revision	2a274ab
Libraries
Qt version	5.15.13
Python version	3.12.3
GDAL/OGR version	3.8.4
PROJ version	9.4.0
EPSG Registry database version	v11.004 (2024-02-24)
GEOS version	3.12.1-CAPI-1.18.1
SQLite version	3.45.1
PostgreSQL client version	16.6 (Ubuntu 16.6-0ubuntu0.24.04.1)
SpatiaLite version	5.1.0
QWT version	6.1.4
QScintilla2 version	2.14.1
OS version	Ubuntu 24.04.1 LTS
Active Python plugins
quick_map_services	0.19.37
QuickWKT	3.1
simplesvg	3.1.1
latlontools	3.7.2
profiletool	4.2.6
processing	2.12.99
grassprovider	2.12.99
db_manager	0.1.20
MetaSearch	0.3.6

Supported QGIS version

• I'm running a supported QGIS version according to the roadmap.

New profile

• I tried with a new QGIS profile

Additional context

I have confirmed that the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY keys and values entered in the protocol credential option variables are the same as entered as QGIS System-level environment variables on successful executions (and used within related Python GDAL code).

I can create separate disposable AWS user access keys to aid in others troubleshooting if needed.

[rouault]

I can create separate disposable AWS user access keys to aid in others troubleshooting if needed.

would be needed. I can just add fine a GeoTIFF from my S3 bucket using AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY keys and values set in the Add raster layer dialog.

[rtmorgan]

@rouault may I email you the bucket, object and same access key info so that I do not include them in the public github issue?

If so, what email address should I use? Thank you.

[rouault]

@rtmorgan even.rouault at spatialys.com

[rtmorgan]

@rouault email on the way from robert.thomas.morgan at gmail.com

[rouault]

@rtmorgan Just tried with your credentials and that works fine. Are you sure you aren't pasting extra leading or trailing spaces in the credential values ?

[rtmorgan]

@rouault It does not appear to have extra leading or trailing spaces in the values. I will type the keys in this evening (thankfully they're not very long) and report back with the results. Thank you again for helping.

[rtmorgan]

@rouault I'm going to test more things, but I believe the 'AWS_PROFILE' environment variable may be the source of the issue.

I started reading through the '/vsis3/' portion of the docs https://gdal.org/en/stable/user/virtual_file_systems.html#vsis3-aws-s3-files to better understand the configuration options and identify what might be set incorrectly or missing.

I noted that the AWS_PROFILE (GDAL >= 3.2) defaults to 'default' but that configuration is not available in the Credential Options drop-down values.

I already have a configured AWS credentials file with a default profile that is used for tasks unrelated to the GIS work and the S3 location.

Because AWS_PROFILE is not available in the AWS S3 protocol Credential Options drop-down values, I could not update it there. I added it to the QGIS System environment and set its value to a new profile I then added to my existing ~/.aws/credentials file with the AWS access key id and AWS secret access key we were troubleshooting with.

With no options set in the Credential Options, I was able to add the AWS S3 raster layer - as I believe it is reading the AWS access key id and AWS secret access key from the AWS credential profile referenced in the QGIS System environment variable AWS_PROFILE.

[rtmorgan]

As it has a default value that can't currently be overridden in the AWS S3 protocol credential options window to support layer-specific credentials, AWS_PROFILE should probably be added to the drop-down lists for the places where VSIS3-related layer configuration options can be selected, like the AWS S3 protocol credential options window we've been discussing.

I could attempt a PR if that would help (but it would be my first to QGIS so it might go slowly).

[rouault]

Will be fixed in GDAL per OSGeo/gdal@4d2e653
You can use the exiting (and deprecated) AWS_DEFAULT_PROFILE in the mean time

[rtmorgan]

thank you - I appreciate this and all you and the maintainers do for the QGIS community.