Skip to content

Refactor Token Caching with CachedTokenSource Wrapper #461

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 50 commits into from
Jun 26, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
2374611
First draft
emmyzhou-db May 28, 2025
5148136
Update test
emmyzhou-db May 28, 2025
78c03a8
Clean up unit tests
emmyzhou-db May 30, 2025
5a37c59
Clean up comments
emmyzhou-db May 30, 2025
f5e4f8a
Add Javadoc to Token.java
emmyzhou-db May 30, 2025
3e65109
Add a token expiry buffer field
emmyzhou-db May 30, 2025
dfce414
Fix for comments
emmyzhou-db Jun 2, 2025
5bd4215
Update tests
emmyzhou-db Jun 3, 2025
93e0baf
Clean up tests
emmyzhou-db Jun 3, 2025
15e221d
Add logging
emmyzhou-db Jun 3, 2025
7589dab
Performance optimization
emmyzhou-db Jun 3, 2025
d97c734
Furter optimizations
emmyzhou-db Jun 3, 2025
105bc99
Add extra token state check in async refresh
emmyzhou-db Jun 3, 2025
b24a0fc
Change LocalDateTime to Instant
emmyzhou-db Jun 3, 2025
6f81b4c
Update parseExpiry in CilTokenSource
emmyzhou-db Jun 4, 2025
daac1b2
Update javadoc
emmyzhou-db Jun 4, 2025
f3d4b8a
Update javadoc
emmyzhou-db Jun 4, 2025
12123a9
Retrigger tests
emmyzhou-db Jun 5, 2025
def03c5
Merge branch 'main' into emmyzhou-db/localdatetime-to-instant
parthban-db Jun 5, 2025
8705ff5
Save progress
emmyzhou-db Jun 6, 2025
fdc50ef
Removed redundant date formattters
emmyzhou-db Jun 6, 2025
70934b2
Change clock supplier to use UTC time
emmyzhou-db Jun 6, 2025
8f19819
Update async tests
emmyzhou-db Jun 7, 2025
333d22e
Update async test
emmyzhou-db Jun 7, 2025
1bd052f
Add support for space separated expiry strings
emmyzhou-db Jun 7, 2025
408f3b4
revert test data
emmyzhou-db Jun 7, 2025
3f06444
Passing all tests
emmyzhou-db Jun 10, 2025
f12024f
Wrapped all token sources in cached token source
emmyzhou-db Jun 10, 2025
c32b752
update names
emmyzhou-db Jun 10, 2025
7e441d6
Small change
emmyzhou-db Jun 10, 2025
35cf2a4
Revert DatabricksConfig
emmyzhou-db Jun 10, 2025
b55a4e8
Merge branch 'emmyzhou-db/localdatetime-to-instant' into emmyzhou-db/…
emmyzhou-db Jun 10, 2025
457d02c
Merge branch 'emmyzhou-db/localdatetime-to-instant' into emmyzhou-db/…
emmyzhou-db Jun 10, 2025
1c99d4b
Merge branch 'emmyzhou-db/localdatetime-to-instant' into emmyzhou-db/…
emmyzhou-db Jun 10, 2025
63d246c
Use Instant with CachedTokenSource
emmyzhou-db Jun 10, 2025
114982e
Update AzureCliCredentials
emmyzhou-db Jun 11, 2025
8417e1b
Revert AzureCliCredentialsProvider
emmyzhou-db Jun 11, 2025
e843dc7
Update variable names
emmyzhou-db Jun 11, 2025
6cc046c
Merge branch 'main' into emmyzhou-db/async_token_cache_wrapper
emmyzhou-db Jun 16, 2025
b774e1a
clean up CachedTokenSource
emmyzhou-db Jun 16, 2025
e846dd7
Rename SystemClockSupplier to UtcClockSupplier
emmyzhou-db Jun 16, 2025
8970e1b
Refactor
emmyzhou-db Jun 17, 2025
e3499a8
Merge branch 'main' into emmyzhou-db/async_token_cache_wrapper
emmyzhou-db Jun 18, 2025
c880323
Clean up
emmyzhou-db Jun 18, 2025
75367bf
Add javadoc to builder class
emmyzhou-db Jun 18, 2025
a6b8df2
Merge branch 'main' into emmyzhou-db/async_token_cache_wrapper
emmyzhou-db Jun 26, 2025
bc66554
Use set instead of with for cached token source builder
emmyzhou-db Jun 26, 2025
3d6c39a
Update names
emmyzhou-db Jun 26, 2025
d61fe31
Update unnecessary wrapping in tests
emmyzhou-db Jun 26, 2025
663801a
make token private
emmyzhou-db Jun 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package com.databricks.sdk.core;

import com.databricks.sdk.core.oauth.CachedTokenSource;
import com.databricks.sdk.core.oauth.OAuthHeaderFactory;
import com.databricks.sdk.core.oauth.Token;
import com.databricks.sdk.core.utils.AzureUtils;
Expand All @@ -19,7 +20,7 @@ public String authType() {
return AZURE_CLI;
}

public CliTokenSource tokenSourceFor(DatabricksConfig config, String resource) {
public CachedTokenSource tokenSourceFor(DatabricksConfig config, String resource) {
String azPath =
Optional.ofNullable(config.getEnv()).map(env -> env.get("AZ_PATH")).orElse("az");

Expand All @@ -35,7 +36,7 @@ public CliTokenSource tokenSourceFor(DatabricksConfig config, String resource) {
List<String> extendedCmd = new ArrayList<>(cmd);
extendedCmd.addAll(Arrays.asList("--subscription", subscription.get()));
try {
return getToken(config, extendedCmd);
return getTokenSource(config, extendedCmd);
} catch (DatabricksException ex) {
LOG.warn("Failed to get token for subscription. Using resource only token.");
}
Expand All @@ -45,14 +46,15 @@ public CliTokenSource tokenSourceFor(DatabricksConfig config, String resource) {
+ "It is recommended to specify this field in the Databricks configuration to avoid authentication errors.");
}

return getToken(config, cmd);
return getTokenSource(config, cmd);
}

protected CliTokenSource getToken(DatabricksConfig config, List<String> cmd) {
CliTokenSource token =
protected CachedTokenSource getTokenSource(DatabricksConfig config, List<String> cmd) {
CliTokenSource tokenSource =
new CliTokenSource(cmd, "tokenType", "accessToken", "expiresOn", config.getEnv());
token.getToken(); // We need this to check if the CLI is installed and to validate the config.
return token;
CachedTokenSource cachedTokenSource = new CachedTokenSource.Builder(tokenSource).build();
cachedTokenSource.getToken(); // Check if the CLI is installed and to validate the config.
return cachedTokenSource;
}

private Optional<String> getSubscription(DatabricksConfig config) {
Expand All @@ -77,16 +79,16 @@ public OAuthHeaderFactory configure(DatabricksConfig config) {
try {
AzureUtils.ensureHostPresent(config, mapper, this::tokenSourceFor);
String resource = config.getEffectiveAzureLoginAppId();
CliTokenSource tokenSource = tokenSourceFor(config, resource);
CliTokenSource mgmtTokenSource;
CachedTokenSource tokenSource = tokenSourceFor(config, resource);
CachedTokenSource mgmtTokenSource;
try {
mgmtTokenSource =
tokenSourceFor(config, config.getAzureEnvironment().getServiceManagementEndpoint());
} catch (Exception e) {
LOG.debug("Not including service management token in headers", e);
mgmtTokenSource = null;
}
CliTokenSource finalMgmtTokenSource = mgmtTokenSource;
CachedTokenSource finalMgmtTokenSource = mgmtTokenSource;
return OAuthHeaderFactory.fromSuppliers(
tokenSource::getToken,
() -> {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package com.databricks.sdk.core;

import com.databricks.sdk.core.oauth.RefreshableTokenSource;
import com.databricks.sdk.core.oauth.Token;
import com.databricks.sdk.core.oauth.TokenSource;
import com.databricks.sdk.core.utils.Environment;
import com.databricks.sdk.core.utils.OSUtils;
import com.fasterxml.jackson.databind.JsonNode;
Expand All @@ -18,7 +18,7 @@
import java.util.List;
import org.apache.commons.io.IOUtils;

public class CliTokenSource extends RefreshableTokenSource {
public class CliTokenSource implements TokenSource {
private List<String> cmd;
private String tokenTypeField;
private String accessTokenField;
Expand Down Expand Up @@ -86,7 +86,7 @@ private String getProcessStream(InputStream stream) throws IOException {
}

@Override
protected Token refresh() {
public Token getToken() {
try {
ProcessBuilder processBuilder = new ProcessBuilder(cmd);
processBuilder.environment().putAll(env.getEnv());
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package com.databricks.sdk.core;

import com.databricks.sdk.core.oauth.CachedTokenSource;
import com.databricks.sdk.core.oauth.OAuthHeaderFactory;
import com.databricks.sdk.core.utils.OSUtils;
import java.util.*;
Expand Down Expand Up @@ -47,8 +48,11 @@ public OAuthHeaderFactory configure(DatabricksConfig config) {
if (tokenSource == null) {
return null;
}
tokenSource.getToken(); // We need this for checking if databricks CLI is installed.
return OAuthHeaderFactory.fromTokenSource(tokenSource);

CachedTokenSource cachedTokenSource = new CachedTokenSource.Builder(tokenSource).build();
cachedTokenSource.getToken(); // We need this for checking if databricks CLI is installed.

return OAuthHeaderFactory.fromTokenSource(cachedTokenSource);
} catch (DatabricksException e) {
String stderr = e.getMessage();
if (stderr.contains("not found")) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ public OAuthHeaderFactory configure(DatabricksConfig config) {
config.getEffectiveAzureLoginAppId(),
idToken.get(),
"urn:ietf:params:oauth:client-assertion-type:jwt-bearer");

return OAuthHeaderFactory.fromTokenSource(tokenSource);
CachedTokenSource cachedTokenSource = new CachedTokenSource.Builder(tokenSource).build();
return OAuthHeaderFactory.fromTokenSource(cachedTokenSource);
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ public OAuthHeaderFactory configure(DatabricksConfig config) {
}
AzureUtils.ensureHostPresent(
config, mapper, AzureServicePrincipalCredentialsProvider::tokenSourceFor);
RefreshableTokenSource inner = tokenSourceFor(config, config.getEffectiveAzureLoginAppId());
RefreshableTokenSource cloud =
CachedTokenSource inner = tokenSourceFor(config, config.getEffectiveAzureLoginAppId());
CachedTokenSource cloud =
tokenSourceFor(config, config.getAzureEnvironment().getServiceManagementEndpoint());

return OAuthHeaderFactory.fromSuppliers(
Expand All @@ -44,29 +44,32 @@ public OAuthHeaderFactory configure(DatabricksConfig config) {
}

/**
* Creates a RefreshableTokenSource for the specified Azure resource.
* Creates a CachedTokenSource for the specified Azure resource.
*
* <p>This function constructs a RefreshableTokenSource instance that fetches OAuth tokens for the
* <p>This function constructs a CachedTokenSource instance that fetches OAuth tokens for the
* given Azure resource. It uses the authentication parameters provided by the DatabricksConfig
* instance to generate the tokens.
*
* @param config The DatabricksConfig instance containing the required authentication parameters.
* @param resource The Azure resource for which OAuth tokens need to be fetched.
* @return A RefreshableTokenSource instance capable of fetching OAuth tokens for the specified
* Azure resource.
* @return A CachedTokenSource instance capable of fetching OAuth tokens for the specified Azure
* resource.
*/
private static RefreshableTokenSource tokenSourceFor(DatabricksConfig config, String resource) {
private static CachedTokenSource tokenSourceFor(DatabricksConfig config, String resource) {
String aadEndpoint = config.getAzureEnvironment().getActiveDirectoryEndpoint();
String tokenUrl = aadEndpoint + config.getAzureTenantId() + "/oauth2/token";
Map<String, String> endpointParams = new HashMap<>();
endpointParams.put("resource", resource);
return new ClientCredentials.Builder()
.withHttpClient(config.getHttpClient())
.withClientId(config.getAzureClientId())
.withClientSecret(config.getAzureClientSecret())
.withTokenUrl(tokenUrl)
.withEndpointParametersSupplier(() -> endpointParams)
.withAuthParameterPosition(AuthParameterPosition.BODY)
.build();

ClientCredentials clientCredentials =
new ClientCredentials.Builder()
.withHttpClient(config.getHttpClient())
.withClientId(config.getAzureClientId())
.withClientSecret(config.getAzureClientSecret())
.withTokenUrl(tokenUrl)
.withEndpointParametersSupplier(() -> endpointParams)
.withAuthParameterPosition(AuthParameterPosition.BODY)
.build();
return new CachedTokenSource.Builder(clientCredentials).build();
}
}
Loading
Loading