Learn how the Azure Metrics Data plane API works: metrics:getBatch

[zmoog]

While testing the new metrics, I found an expected behavior: the getBatch endpoint from the Azure Metrics Data plane API returns different results for the same resource depending on whether I'm getting value for single or multiple resources.

For example, if I get the metric value for one resource:

# pseudo code
resourceIDs = [
  "resource-1"
]

get_metric_values(resourceIDs)

I get three time-series values.

However, if I get the same metric values for resource-1 and other nice resources:

# pseudo code
resourceIDs = [
  "resource-1",
  "resource-2",
  "resource-3",
  "resource-4",
  "resource-5",
  "resource-6",
  "resource-7",
  "resource-8",
  "resource-9",
  "resource-10",
]

get_metric_values(resourceIDs)

I expect to see the same results but get less than three results for resource-1 (often zero).

This difference is probably due to me doing something wrong, but I don't get what. Let's find out.

[zmoog]

I created a gist with the Go code I was testing at https://gist.github.com/zmoog/fcede6fcbe5ba11f9275c40a58eea38d

[zmoog]

I'm calling the underlying endpoint to simplify things, skipping the beta SDK.

1. get a token
2. test with a single resource
3. test with ten resources

get a token

az account get-access-token -t <your-tenant-id-here> | jq .accessToken | pbcopy    

test with a single resource

$ curl -X POST "https://eastus2.metrics.monitor.azure.com/subscriptions/<YOUR-SUBSCRIPTION-HERE>/metrics:getBatch?api-version=2023-10-01&metricnamespace=Microsoft.KeyVault/vaults&starttime=2024-04-16T07:18:13.001Z&endtime=2024-04-16T07:19:13.001Z&interval=PT1M&metricnames=ServiceApiResult&filter=ActivityType%20eq%20%27%2A%27%20AND%20ActivityName%20eq%20%27%2A%27%20AND%20StatusCode%20eq%20%27%2A%27%20AND%20StatusCodeClass%20eq%20%27%2A%27&aggregation=Count" \
    -H "Authorization: bearer ${TOKEN}" \
    -H "Content-Type: application/json" \
    --data @single.json | jq '.values[].value[] | .timeseries | length'

3

test with ten resources

$ curl -X POST "https://eastus2.metrics.monitor.azure.com/subscriptions/<YOUR-SUBSCRIPTION-HERE>/metrics:getBatch?api-version=2023-10-01&metricnamespace=Microsoft.KeyVault/vaults&starttime=2024-04-16T07:18:13.001Z&endtime=2024-04-16T07:19:13.001Z&interval=PT1M&metricnames=ServiceApiResult&filter=ActivityType%20eq%20%27%2A%27%20AND%20ActivityName%20eq%20%27%2A%27%20AND%20StatusCode%20eq%20%27%2A%27%20AND%20StatusCodeClass%20eq%20%27%2A%27&aggregation=Count" \
    -H "Authorization: bearer ${TOKEN}" \
    -H "Content-Type: application/json" \
    --data @multiple.json | jq '.values[].value[] | .timeseries | length'

# first run
1
2
0
0
2
0
2
2
0
1

# second run
1
1
1
1
0
0
2
2
0
2

Here are the two JSON files content:

# single.json
{
    "resourceids": [
        "/subscriptions/<YOUR-SUBSCRIPTION-HERE>/resourceGroups/mbranca-az-scalability-kv-r10/providers/Microsoft.KeyVault/vaults/mbrancar10s008"
    ]
}

# multiple.json
{
    "resourceids": [
      "/subscriptions/YOUR-SUBSCRIPTION-HERE/resourceGroups/mbranca-az-scalability-kv-r10/providers/Microsoft.KeyVault/vaults/mbrancar10s009",
      "/subscriptions/YOUR-SUBSCRIPTION-HERE/resourceGroups/mbranca-az-scalability-kv-r10/providers/Microsoft.KeyVault/vaults/mbrancar10s003",
      "/subscriptions/YOUR-SUBSCRIPTION-HERE/resourceGroups/mbranca-az-scalability-kv-r10/providers/Microsoft.KeyVault/vaults/mbrancar10s004",
      "/subscriptions/YOUR-SUBSCRIPTION-HERE/resourceGroups/mbranca-az-scalability-kv-r10/providers/Microsoft.KeyVault/vaults/mbrancar10s005",
      "/subscriptions/YOUR-SUBSCRIPTION-HERE/resourceGroups/mbranca-az-scalability-kv-r10/providers/Microsoft.KeyVault/vaults/mbrancar10s007",
      "/subscriptions/YOUR-SUBSCRIPTION-HERE/resourceGroups/mbranca-az-scalability-kv-r10/providers/Microsoft.KeyVault/vaults/mbrancar10s008",
      "/subscriptions/YOUR-SUBSCRIPTION-HERE/resourceGroups/mbranca-az-scalability-kv-r10/providers/Microsoft.KeyVault/vaults/mbrancar10s010",
      "/subscriptions/YOUR-SUBSCRIPTION-HERE/resourceGroups/mbranca-az-scalability-kv-r10/providers/Microsoft.KeyVault/vaults/mbrancar10s001",
      "/subscriptions/YOUR-SUBSCRIPTION-HERE/resourceGroups/mbranca-az-scalability-kv-r10/providers/Microsoft.KeyVault/vaults/mbrancar10s002",
      "/subscriptions/YOUR-SUBSCRIPTION-HERE/resourceGroups/mbranca-az-scalability-kv-r10/providers/Microsoft.KeyVault/vaults/mbrancar10s006"
    ]
  }

[zmoog]

I am starting to increase the number of resources to see if there's a threshold in this behavior.

I repeated each request at least three times.

$ curl -X POST "https://eastus2.metrics.monitor.azure.com/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/metrics:getBatch?api-version=2023-10-01&metricnamespace=Microsoft.KeyVault/vaults&starttime=2024-04-16T07:18:13.001Z&endtime=2024-04-16T07:19:13.001Z&interval=PT1M&metricnames=ServiceApiResult&filter=ActivityType%20eq%20%27%2A%27%20AND%20ActivityName%20eq%20%27%2A%27%20AND%20StatusCode%20eq%20%27%2A%27%20AND%20StatusCodeClass%20eq%20%27%2A%27&aggregation=Count" \
    -H "Authorization: bearer ${TOKEN}" \
    -H "Content-Type: application/json" \
    --data @single.json | jq '.values[].value[] | .timeseries | length'

3


$ curl -X POST "https://eastus2.metrics.monitor.azure.com/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/metrics:getBatch?api-version=2023-10-01&metricnamespace=Microsoft.KeyVault/vaults&starttime=2024-04-16T07:18:13.001Z&endtime=2024-04-16T07:19:13.001Z&interval=PT1M&metricnames=ServiceApiResult&filter=ActivityType%20eq%20%27%2A%27%20AND%20ActivityName%20eq%20%27%2A%27%20AND%20StatusCode%20eq%20%27%2A%27%20AND%20StatusCodeClass%20eq%20%27%2A%27&aggregation=Count" \
    -H "Authorization: bearer ${TOKEN}" \
    -H "Content-Type: application/json" \
    --data @two.json | jq '.values[].value[] | .timeseries | length'

3
3


$ curl -X POST "https://eastus2.metrics.monitor.azure.com/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/metrics:getBatch?api-version=2023-10-01&metricnamespace=Microsoft.KeyVault/vaults&starttime=2024-04-16T07:18:13.001Z&endtime=2024-04-16T07:19:13.001Z&interval=PT1M&metricnames=ServiceApiResult&filter=ActivityType%20eq%20%27%2A%27%20AND%20ActivityName%20eq%20%27%2A%27%20AND%20StatusCode%20eq%20%27%2A%27%20AND%20StatusCodeClass%20eq%20%27%2A%27&aggregation=Count" \
    -H "Authorization: bearer ${TOKEN}" \
    -H "Content-Type: application/json" \
    --data @three.json | jq '.values[].value[] | .timeseries | length' 

3
3
3


$ curl -X POST "https://eastus2.metrics.monitor.azure.com/subscriptions/12cabcb4-86e8-404f-a3d2-1dc9982f45ca/metrics:getBatch?api-version=2023-10-01&metricnamespace=Microsoft.KeyVault/vaults&starttime=2024-04-16T07:18:13.001Z&endtime=2024-04-16T07:19:13.001Z&interval=PT1M&metricnames=ServiceApiResult&filter=ActivityType%20eq%20%27%2A%27%20AND%20ActivityName%20eq%20%27%2A%27%20AND%20StatusCode%20eq%20%27%2A%27%20AND%20StatusCodeClass%20eq%20%27%2A%27&aggregation=Count" \
    -H "Authorization: bearer ${TOKEN}" \
    -H "Content-Type: application/json" \
    --data @four.json | jq '.values[].value[] | .timeseries | length'


# run 1
2
3
2
3

# run 2
2
3
3
2

# run 3
2
3
3
2

[zmoog]

Starting with four resources, the number of results starts changing at each call.