Does aioboto3 Support Authentication with EC2 IAM Roles?

[higuhigu-lb]

Describe the bug
While using the boto3 library, I can authenticate and interact with AWS services using the IAM role assigned to my EC2 instance. However, when attempting to perform similar operations with s3fs, I encounter errors. Below is the code snippet demonstrating the behavior with boto3 and the corresponding error with s3fs.

Environment:

• Python Version: 3.10
• OS name and version: ubuntu
• s3fs-2024.3.1

Sample code

import boto3

# Create boto3 client
s3 = boto3.client('s3')

# List contents of a bucket
response = s3.list_objects_v2(Bucket='your-bucket-name')
print(response)

import s3fs
fs = s3fs.S3FileSystem(anon=False)
# Attempt to list the same bucket with s3fs, expecting to use the same IAM role
print(fs.ls('your-bucket-name'))

Error log

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/s3fs/core.py", line 720, in _lsdir
    async for c in self._iterdir(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/s3fs/core.py", line 770, in _iterdir
    async for i in it:
  File "/home/ubuntu/.local/lib/python3.10/site-packages/aiobotocore/paginate.py", line 30, in __anext__
    response = await self._make_request(current_kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/aiobotocore/client.py", line 408, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: No AWSAccessKey was presented.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "/home/ubuntu/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
  File "/home/ubuntu/.local/lib/python3.10/site-packages/s3fs/core.py", line 1005, in _ls
    files = await self._lsdir(path, refresh, versions=versions)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/s3fs/core.py", line 733, in _lsdir
    raise translate_boto_error(e)
PermissionError: No AWSAccessKey was presented.

[martindurant]

The truth is, I don't know how to do this, but I assume it must be possible. I recommend you play with the botocore API, and find what works. From there, it should be easy to phrase this correctly for s3fs. Since everything is supposed to work via instance metadata, perhaps it's worth looking on the [aio]botocore repos to see if anyone is experiencing something similar.

[emileten]

I have been having the same issue and couldn't find a solution so far.

[Detectronic-AH]

Me too.