Add support for access-token secrets

[mmaitre314]

• Implement Support for specifying token directly #20
• Add a new ACCESS_TOKEN secret provider
• Add test
• Include build instructions from the main duckdb repo in README.md

[samansmink]

Hey @mmaitre314 thanks for the PR, looks great!

To fix the CI failure, could you add the ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true env variable to the functional test CI jobs (like here https://github.com/duckdb/extension-ci-tools/blob/ebf18ed49b11e656adc20d721bf7dac2de15d439/.github/workflows/_extension_distribution.yml#L155)
Our Linux CI is a little broken at the moment and while we refactor a few things we need to have this workaround.

[mmaitre314]

I added ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION in a couple of places but I am not very confident in where I added that... If I understand correctly, patch azure_http_state.patch is not getting applied and the code tries to include duckdb/common/http_state.hpp which does not exist anymore.

[samansmink]

Ah I see, sorry I missed that Azures CI is running against DuckDB's main branch. Normally we have CI target latest stable and let duckdb/duckdb CI take care of catching changes breaking compatiblity.

I've applied the patch to your branch, this should fix ci here. Then we can bump azure's version in duckdb/duckdb ci and remove the patch.

[mmaitre314]

The formatting of HTTP stats changed a bit so I updated the Regex in the test to match.

Before: ss << "││ HTTP Stats: ││\n";

Now: ss << "││" + QueryProfiler::DrawPadded("Azure HTTP Stats", TOTAL_BOX_WIDTH - 4) + "││\n";

[samansmink]

thanks @mmaitre314!

[mmaitre314]

Looking into updating docs as next step, should I wait until a new version of DuckDB is released?

[Lyndon1994]

Hi @mmaitre314
Please consider this scenario:

00:00, set access token in secret (assume One hour valid)
00:54, run a sql, with current token （token1)
00:55, refresh token to token2
01:01, token1 expired, the sql will be failed.

Will the sql execution detect the token expiration or re-fetch the token?

[mmaitre314]

@Lyndon1994 -- this does not automatically handle token refresh and the caller would need to ensure the token is valid for long enough.

[Lyndon1994]

Yes, is there any plan to support this feature?

[mmaitre314]

Not that I am aware of. This is an advanced scenario and I am not sure simplifying further would be worth the work (track token lifetime, add a callback to get a new access token from the caller, etc.). Other forms of secrets are more user friendly in that respect.

[Lyndon1994]

We got this issue in the Fabric Notebook, because we can only use access token to access OneLake.

[Lyndon1994]

Is it easy to read environment variable to get access token?

[mmaitre314]

In Fabric Notebooks I believe notebookutils.credentials.getToken('storage') [doc] would be used to get access tokens, and likely not environment variables. I have limited experience with Fabric though so I'll ask around to see if there is some better guidance.