[Bug]: SFTP on S3 does not allow uploading new file and listing subdirectory when s3:ListObject is restricted on s3:prefix condition

[ibotty]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration issue.
• This issue is not already reported on Github (I've searched it).

Bug description

SFTP on a S3-compatible Ceph RadosGW with a prefix works great, but fails when using a bucket policy that restricts the s3:ListBucket action. See an example policy below.

According to the documentation if one does not have the s3:ListBucket permissions a non-existent file gets a 403 status code instead of an 404.

This trips up sftpgo in that it can't upload files into a new file name, copying into an existing filename works fine.

Also directory listings (ls subdir/) do not seem to work, but I have not debugged that.

{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"allow-list",
      "Effect":"Allow",
      "Action": [
        "s3:ListBucket",
        "s3:ListBucketVersions"
      ],
      "Principal": {
        "AWS": [
          "arn:aws:iam:::user/my-user"
        ]
      },
      "Condition" : {
        "StringLike" : {
          "s3:prefix": [
            "my-path/*"
          ]
        }
      },
      "Resource": ["arn:aws:s3:::my-bucket"]
    },
    {
      "Sid":"allow-get-put",
      "Effect":"Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Principal": {
        "AWS": [
          "arn:aws:iam:::user/my-user"
        ]
      },
      "Resource": [
        "arn:aws:s3:::my-bucket/my-path/*"
      ]
    }
 ]
}

Steps to reproduce

1. setup s3-bucket and sftpgo with my-path prefix as usual
2. s3cmd setpolicy my-bucketpolicy.json s3://my-bucket
3. sftp my-sftpgo and

sftp> put a
Uploading a to /a
dest open "/a": Permission denied

Expected behavior

sftp> put a
Uploading a to /a
a                        100%    4     0.1KB/s   00:00

SFTPGo version

SFTPGo 2.6.3-cf3e1d3e-2024-11-15T16:52:00Z +metrics +azblob +gcs +s3 +bolt +mysql +pgsql +sqlite +unixcrypt +portable

Data provider

memory

Installation method

Community Docker image

Configuration

This happens with and without SFTPGO_COMMON__SETSTAT_MODE=1.

Relevant log output

{
  "level": "error",
  "time": "2024-11-21T15:15:20.694",
  "sender": "SFTP",
  "connection_id": "SFTP_c6bf06016dfb63b365834cc39bd416c60bdbbc6344fc61507246694ae49e535d_3",
  "message": "error performing file stat \"my-path/a\": operation error S3: HeadObject, https response error StatusCode: 403, RequestID: tx00000e5b2c463602f2b5c-00673f4e88-144709124-s3, HostID: , api error Forbidden: Forbidden"
}

### What are you using SFTPGo for?

Private user, home usecase (home backup/VPS), Medium business

### Additional info

I would certainly like to fix this myself, but I didn't find where this is coming from.  Any hints?  I might be able to contribute a pull request.