Aggregator transform on count metrics don't behave as expected

[timtylin]

Using version 0.42.0

I'm ingesting millions of of log events that I'm transforming into +1 count metrics. These events span over 1 hour. The metric conversion part went smoothly, but then I was hoping to aggregate them into 10 second bins before pushing to the metric server.

However, during testing I couldn't seem to get the metric aggregation to respect the interval_ms setting. I've expected set it to 10_000, but that seemed to have aggregated everything into a single metric. Further playing around with the metric showed that the value I enter into interval_ms seem to be treated as "minutes" instead of "milliseconds". For example, if I input a value of 2 for interval_ms, all my metrics get aggregated into 2 minute bins. Is this a bug?

Here's the relevant part of my config

[sources.in]
type = "stdin"

[transforms.alb]
type = "remap"
inputs = ["in"]
source = '''
  . = parse_aws_alb_log!(.message)
  .timestamp = parse_timestamp!(.request_creation_time, "%+")
'''

[transforms.alb_metric]
type = "log_to_metric"
inputs = [ "alb" ]

  [[transforms.alb_metric.metrics]]
  type = "counter"
  field = "elb_status_code"
  name = "load_balancer_request_response"
  namespace = "test"

    [transforms.alb_metric.metrics.tags]
    status = "{{elb_status_code}}"
    target_group = "{{target_group_arn}}"


[transforms.aggregated_alb_metric]
type = "aggregate"
inputs = [ "alb_metric"]
interval_ms = 10_000

[jszwedko]

I think the confusion here may lie with how interval_ms works. It aggregates based on real-time windows rather than using the timestamps of the metrics. That is: if you set 10_000, it will aggregate incoming metrics for 10 seconds and then flush. I think you are seeing a single metric when setting it to 10_000 because it is processing all of the input much more quickly than that.

[jszwedko]

I think #14015 may describe the behavior you are looking for.

[timtylin]

@jszwedko Ah I see now, thanks for clearing that up. I guess I misinterpreted what I'm seeing due to the parsing happening to take ~60ms for the whole batch

If I'm reading your linked ticket correctly, aggregating based on event body timestamp is still not a native feature and the recommendation is still to use Lua transform for this (I'm assuming VRL can't do this either because it's stateless). Is that a correct summary of the situation as it stand today?

[jszwedko]

Yeah, unfortunately I think lua is still the best option at the moment. It'd be useful to have a native feature.