Is there a way to configure/use AWS Web Identity Token Provider for S3 access?

[jeffreyeriksondg]

We're attempting to run Nessie version 0.91.3 on our AWS EKS cluster. We have our pods configured so that they're associated with an IAM role with all the permissions they need, and they have the AWS_WEB_IDENTITY_TOKEN_FILE environment variable set. However, in the Nessie Helm chart, I don't see a way to configure Nessie to use the web identity token.

What I've tried so far:

• assuming S3 access would follow the AWS SDK default credential provider chain (which includes using the web identity token), I left S3 storage options unconfigured, which resulted in an exception Secret reference to S3 access key ID is not defined

• configuring storage.s3.defaultOptions.accessKeySecret as an empty object resulted in the same error

• configuring ASSUME_ROLE hoping perhaps STS interactions would be able to use the default credential provider chain resulted in the same error

Looking into the code (here and here), it appears that configuring basic credentials are required for S3 interactions, and the default credentials chain is ignored. But it's very likely I'm missing something.

When we're running Spark jobs, for example, we can configure com.amazonaws.auth.WebIdentityTokenCredentialsProvider which knows how to use that environment variable for S3 credentials. And in other cases, applications will use the default credentials chain if nothing particular is configured. Is there a way we can configure Nessie to do something similar?

What's interesting is that we didn't have this problem with the older version of Nessie we were using (v 0.79.0), perhaps b/c we're trying to use the Iceberg REST API capabilities now?

[dimas-b]

Could #8987 help in this case?

[jeffreyeriksondg]

Yes, I believe it would . . . I searched through issues before posting, and didn't immediately find anything, but I hadn't looked through current PRs :/ Thanks!

[dimas-b]

What's interesting is that we didn't have this problem with the older version of Nessie we were using (v 0.79.0), perhaps b/c we're trying to use the Iceberg REST API capabilities now?

Correct. When using the "core" Nessie API, only the client (Iceberg) is accessing S3. When Nessie is accessed via the Iceberg REST API, the Nessie Server itself has to access S3.

[jeffreyeriksondg]

Looks like the referenced PR has been merged . . . thanks for the work on that! I'm guessing if I were to use the ghcr.io/projectnessie/nessie-unstable:0.91.4 image once it's available with the changes, I might be able to test that change on our experimental systems?

[jeffreyeriksondg]

It looked like the "Publish in-development builds from main" action ran periodically, so I was hoping that maybe this change would be available for testing. I might carve out some bandwidth in the next few days to try building locally . . . if you can't tell, my group is excited about what Nessie offers and are itching to get this working :)

[snazy]

Yup - snapshots are published every 12 hours during weekdays

[snazy]

See https://github.com/projectnessie/nessie/actions/workflows/snapshot-publish.yml

[jeffreyeriksondg]

Perfect. I'll give it a try when the next image is available.

[jeffreyeriksondg]

Looks like the changes from #8987 are working very well! I had to do this in my values.yaml, since it doesn't look like there's an "unstable" Helm chart published:

advancedConfig:
  nessie.catalog.service.s3.default-options.server-auth-mode: APPLICATION_GLOBAL

but once the change is released officially, we can easily update to the proper configuration with Helm.

Thanks for the responsiveness!