[BUG] Azure.Identity.AuthenticationFailedException : AzurePowerShellCredential authentication failed: String '14-7-2021 06:36:10 +00:00' was not recognized as a valid DateTime

[dcanamares]

Describe the bug
I'm running some integration tests on Azure Pipelines, it was working fine until 13 of July (code was released at the beginning of July). These tests connect to an azure subscription to get some values from an Azure KeyVault. I don't send any datetime and I didn't see any way to send a date or a specific format. The issue happens running the tests in the pipeline but it doesn't happen running the tests on a local machine.

I ran a Get-Culture command and the value is:
LCID Name DisplayName

1043 nl-NL Dutch (Netherlands)

Expected behavior
Connect to key vault and retrieve the secrets.

Actual behavior (include Exception or Stack Trace)
From day 1 to day 12 of a month pipeline can connect to Azure keyvault and gets the secrets stored, from day 13 to 31 of a month pipelines throws and exception because the format of the date is dd-MM-yyyy.

Error Message:
Azure.Identity.AuthenticationFailedException : AzurePowerShellCredential authentication failed: String '14-7-2021 06:36:10 +00:00' was not recognized as a valid DateTime.
---- System.FormatException : String '14-7-2021 06:36:10 +00:00' was not recognized as a valid DateTime.
Stack Trace:
at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
at Azure.Identity.AzurePowerShellCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
at Azure.Identity.AzurePowerShellCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
at Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources, TokenRequestContext requestContext, Boolean async, CancellationToken cancellationToken)
at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex)
at Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
at Azure.Identity.DefaultAzureCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueFromCredentialAsync(TokenRequestContext context, Boolean async, CancellationToken cancellationToken)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetHeaderValueAsync(HttpMessage message, TokenRequestContext context, Boolean async)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AuthenticateAndAuthorizeRequestAsync(HttpMessage message, TokenRequestContext context)
at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.AuthorizeRequestOnChallengeAsyncInternal(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.HttpPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken) at Azure.Security.KeyVault.KeyVaultPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken) at Azure.Security.KeyVault.KeyVaultPipeline.GetPageAsync[T](Uri firstPageUri, String nextLink, Func1 itemFactory, String operationName, CancellationToken cancellationToken)
at Azure.Core.PageResponseEnumerator.FuncAsyncPageable1.AsPages(String continuationToken, Nullable1 pageSizeHint)
at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.GetResult(Int16 token) at Azure.AsyncPageable1.GetAsyncEnumerator(CancellationToken cancellationToken)
at Azure.AsyncPageable1.GetAsyncEnumerator(CancellationToken cancellationToken) at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore1.GetResult(Int16 token)
at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.LoadAsync()
at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.LoadAsync()
at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.Load()
at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()

To Reproduce
Configuration :

            var configBuilder = new ConfigurationBuilder()
                .AddJsonFile("testsettings.json", true)
                .AddEnvironmentVariables();

            var builtConfig = configBuilder.Build();

            Configuration = configBuilder
                .AddAzureKeyVault(new Uri(builtConfig["key_vault_azure_uri"]), new DefaultAzureCredential())
                .Build();

To get a secret:
var secret= Configuration["secret_key"]

Environment:

• The project is on Net Core 2.2
• Azure.Identity 1.4.0, Azure.Security.KeyVault.Secrets 4.2.0,
• Pipelines run in a hosted agent with Windows Server 2019
• Visual Studio PRO 2019 16.8.1.0

[jsquire]

Thank you for your feedback. Tagging and routing to the team members best able to assist.

[brwilkinson]

@christothes this update was merged to identity 1.4.1 is that correct?

I seem to have some further reports of this when using:

get-culture

LCID             Name             DisplayName
----             ----             -----------
8192             en-DK            English (Denmark)

Unhandled exception: Azure.Identity.AuthenticationFailedException: The ChainedTokenCredential failed due to an unhandled exception: AzurePowerShellCredential authentication failed: String '13/10/2021 06.54.13 +00:00' was not recognized as a valid DateTime.
 ---> Azure.Identity.AuthenticationFailedException: AzurePowerShellCredential authentication failed: String '13/10/2021 06.54.13 +00:00' was not recognized as a valid DateTime.

[dcanamares]

@brwilkinson No, It was merged in 1.5.0 beta 3 or beta 4.

[brwilkinson]

@dcanamares thank you.

[dcanamares]

@brwilkinson I was testing 1.5.0 and now I've got an exception (only happens with 1.5.0 not with 1.4.0), I've just created a new issue #24688

[JohnGalt1717]

Pretty sure that this fix is what broke culture invariant mode in .NET apps. Please address or revert as you broke a HUGE portion of .net apps that use docker as a result of this.

[christothes]

@JohnGalt1717 - do you have any details on the issue you are referring to? Please provide a stack trace and any relevant details.

If this is not the same issue and stack as described in the original description for this issue, please open a new issue and we'll take a look.

[JohnGalt1717]

@christothes Azure/azure-storage-net#1050

What I'm saying is that when you fixed this issue you broke essentially all docker containers. This needs to be reverted and you need to find a better way that doesn't require use of cultures to format this.

[christothes]

What I'm saying is that when you fixed this issue you broke essentially all docker containers. This needs to be reverted and you need to find a better way that doesn't require use of cultures to format this.

I'd like to help, but I'm not following how all docker containers were broken by having the AzurePowerShellCredential use the current culture when parsing a DateTimeOffset returned by Get-AzAccessToken. Are you possibly referring to a different fix? I'm also not clear how the storage issue linked above is related since Azure.Storage.Blob does not take a direct dependency on Azure.Identity.

It would be helpful if you could provide specific information about the break that now occurs because of this fix including environment details and ideally repro steps.

[JohnGalt1717]

Because the default for all .net containers is invariant culture only, and if you don't do this, you increase the size of your containers by 30-40%. (aside from breaking it) And of course you then have to install the dependencies in your container which are intentionally left out by the .net containers for this very reason.

All of azure storage (like just about everything else azure, depends on Azure Identity. So when you added this parsing based on culture, you broke everything that uses Azure Identity authentication to get a token.

[christothes]

Hi @JohnGalt1717 -
The storage library you are referencing in the referenced issue doesn't appear to be the latest version (it mentions Microsoft.Azure.Storage.* rather than Azure.Storage.*). The Azure.Identity package is not a direct dependency for either package nor does the stack in that issue involve Azure.Identity.

However, You'll find references to Culture in many places besides this specific fix https://grep.app/search?q=CultureInfo.CurrentCulture&filter[repo][0]=Azure/azure-sdk-for-net&filter[path][0]=sdk/ so reverting this won't be of much help if your environment requires strict invariant culture.

[JohnGalt1717]

@christothes well the #1 issue you guys have is that there are about a billion libraries all getting maintained for everything Azure and you don't even note in the nuget release what their status is. So there's that.

But by using culture in your stuff you're making all Docker-based (and thus K8s) use massively more memory and storage for no reason. I'd strongly advise that you fix your libraries and eliminate the need for culture references. Specifically, in Azure.Identity, culture should be irrelevant as you should be returning UTC and ICT dates and times, not things that are culture specific.

This use of culture in these is harming .NET's adoption specifically because you're making it bloat for no reason on Docker and K8s which is the defacto standard at this point for any serious, scaling system.