Skip to content

Commit

Permalink
kafka: remove shard aggregation on consumer group metrics
Browse files Browse the repository at this point in the history
Currently we only aggregate on the `shard` label in the consumer group
metrics. This aggregation doesn't result in a reduced number of metric
series. It does, however, cause seastar to use the
`metric_aggregate_by_labels` class to aggregate the metrics.

This class uses a contiguous map type that isn't easily replaced and in
cases where there are many consumer groups, kafka topics, and partitions
there is enough unique label sets to cause an oversized allocation in
the continguous map.

By removing the `shard` label from aggregation though we avoid the code
path where `metric_aggregate_by_labels` is used and hence avoid the
oversized allocation.

(cherry picked from commit ca8cd1e)
  • Loading branch information
ballard26 authored and vbotbuildovich committed Oct 3, 2024
1 parent 8607c01 commit 8dab9d1
Showing 1 changed file with 6 additions and 9 deletions.
15 changes: 6 additions & 9 deletions src/v/kafka/group_probe.h
Original file line number Diff line number Diff line change
Expand Up @@ -78,11 +78,10 @@ class group_offset_probe {
_public_metrics.add_group(
prometheus_sanitize::metrics_name("kafka:consumer:group"),
{sm::make_gauge(
"committed_offset",
[this] { return _offset; },
sm::description("Consumer group committed offset"),
labels)
.aggregate({sm::shard_label})});
"committed_offset",
[this] { return _offset; },
sm::description("Consumer group committed offset"),
labels)});
}

private:
Expand Down Expand Up @@ -130,15 +129,13 @@ class group_probe {
"consumers",
[this] { return _members.size(); },
sm::description("Number of consumers in a group"),
labels)
.aggregate({sm::shard_label}),
labels),

sm::make_gauge(
"topics",
[this] { return _offsets.size(); },
sm::description("Number of topics in a group"),
labels)
.aggregate({sm::shard_label})});
labels)});
}

private:
Expand Down

0 comments on commit 8dab9d1

Please sign in to comment.