Azure Gen2 broken on latest

[a10y]

Describe the bug

I've been investigating some Vortex queries on Azure Blob Store Gen2. I've created a simple storage account, populated some files, and was just trying to list the bucket on 0.12

Cargo.toml:

[package]
name = "rustazuretest"
version = "0.1.0"
edition = "2024"

[dependencies]
tokio = { version = "1", features = ["full"] }
object_store = { version = "=0.12", features = ["azure"] }
futures = "0.3"

main.rs

use futures::StreamExt;
use object_store::azure::MicrosoftAzureBuilder;
use object_store::ObjectStore;

#[tokio::main]
async fn main() {
    let account = "vortexicebergsummit25";
    let access_key = "REDACTED";
    let container_name = "tpch";

    let azure = MicrosoftAzureBuilder::new()
        .with_account(account)
        .with_access_key(access_key)
        .with_container_name(container_name)
        .build()
        .expect("azure object store");

    let mut objects = azure.list(None);
    while let Some(next) = objects.next().await {
        println!("Object: {:?}", next.unwrap());
    }
}

On 0.12 this results in an error:

called `Result::unwrap()` on an `Err` value: Generic { store: "MicrosoftAzure", source: ListRequest { source: RetryError { method: GET, uri: Some(https://vortexicebergsummit25.blob.core.windows.net/tpch?&restype=container&comp=list), retries: 0, max_retries: 10, elapsed: 61.271375ms, retry_timeout: 180s, inner: Status { status: 403, body: Some("\u{feff}<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:1fe66e1c-a01e-0000-2a10-a476a8000000\nTime:2025-04-02T20:45:18.2401813Z</Message><AuthenticationErrorDetail>The MAC signature found in the HTTP request 'REDACTED' is not the same as any computed signature. Server used following string to sign: 'GET\n\n\n\n\n\nWed, 02 Apr 2025 20:45:18 GMT\n\n\n\n\n\nx-ms-version:2023-11-03\n/vortexicebergsummit25/tpch\n:\ncomp:list\nrestype:container'.</AuthenticationErrorDetail></Error>") } } } }
stack backtrace:
...

However, if I forcibly pin to 0.11.2, it works

[package]
name = "rustazuretest"
version = "0.1.0"
edition = "2024"

[dependencies]
tokio = { version = "1", features = ["full"] }
object_store = { version = "=0.11.2", features = ["azure"] }
futures = "0.3"

[alamb]

Here are the diff between the two releases:

• apache/arrow-rs@object_store_0.11.2...object_store_0.12.0

[tustvold]

I am afraid I no longer have access to an Azure setup and so no way to debug this, however, my guess would be some change in the headers passed to the signing process stemming from adding the HttpClient abstraction. Most likely either Host or Content-Length.

I think the key questions would be:

• Is it just listing or other methods impacted
• Is it specific to a Gen2 buckets
• Is it specific to a credential type

[a10y]

I did a bit of manual bisecting of any commits that touched src/azure, commit b4b7fd5b7b03b551 seems to be the first one that fails the test in OP. Haven't dug into the changeset yet

[alamb]

I did a bit of manual bisecting of any commits that touched src/azure, commit b4b7fd5b7b03b551 seems to be the first one that fails the test in OP. Haven't dug into the changeset yet

I looked up that commit and it is apache/arrow-rs@b4b7fd5b7b03b551 , which came from

• Decouple ObjectStore from Reqwest arrow-rs#7183

[AdamGS]

I'm looking into it, hopefully I'll have something by end of day.