Generate a new metric from the incoming metrics

[omarghader]

Hello,
I have a use case where I need to count the incoming metrics and generate a new metric with the label "service_namespace".
Based on the docs, I have found only 2 possible solutions:

• Use metric_to_log -> log_to_metric
• Generate metric using Lua code

The problem: these 2 solutions works well, but they are slow or needs a lot more CPU.
Is there any other alternative to generate this metric? Thanks

Here is the code of the 2 solutions I have

Solution 1

      transforms:
        metric_to_log:
          type: metric_to_log
          inputs:
            - prometheusreceiver

        metrics_counter_by_service_namespace:
          type: log_to_metric
          inputs:
            - metric_to_log
          metrics:
            - type: counter
              field: name
              name: count_events_total
              tags:
                service_namespace: {{ .tags.service_namespace }}

Solution 2

transforms:
  metrics_counter:
    type: lua
    inputs:
      - prometheusreceiver
    version: "2"
    hooks:
      init: init
      process: process
      shutdown: shutdown
    timers:
      - interval_seconds: 30
        handler: timer_handler
    source: |-
      function init()
        count = {}
      end

      function process(event)
        if event.metric.tags.service_namespace then
          if not count[event.metric.tags.service_namespace] then
            count[event.metric.tags.service_namespace] = 0
          end
          count[event.metric.tags.service_namespace] = count[event.metric.tags.service_namespace] + 1
        end
      end

      function timer_handler(emit)
        sendMetrics(emit, count)
      end

      function shutdown(emit)
        sendMetrics(emit, count)
      end

      function sendMetrics(emit, count)
        for k,v in pairs(count) do
          metric = {
            name = "count_events_total",
            kind = "incremental",
            timestamp = os.date("!*t"),
            counter = {
              value = v
            },
            tags = {
              service_namespace= k,
            }
          }
          emit({ metric = metric })
          count[k] = 0
        end
      end

[jszwedko]

I think the aggregate transform can be used for this like:

type: aggregate
mode: count
inputs: [...]

That transform should count all of the metrics flowing through it and emit the count as a new metric.

This was added in a recent version of Vetor: I believe v0.41.0.

[omarghader]

Thanks for your answer @jszwedko, I think this aggregate type aggregates the metrics each x seconds but the metric's name and labels stays the same, It's not what I'm looking for.
What I really want is to create a completely new metric with only some specific labels, not all of them. Thank you

[jszwedko]

Ah, gotcha. I think solution (1) you have above is what I would reach for.

[omarghader]

Unfortunately, the solution 1 is slower than the solution 2 or needs more resources
On small amount of metrics, it looks the same, but for 1 Million/s metrics, I have done a test:

• Solution 1: needs 39 replicas with 3 cpus
• Solution 2: needs between 33->35 replicas with 3 cpus

[jszwedko]

Interesting, that is a bit surprising given lua is usually the slowest part of any pipeline.

[omarghader]

Just for info, on kuberenetes solution 2 (lua) required less replicas (with 3 cpu/pod).
However on a VM, the solution 1 (native vector) was quicker indeed.

The kuberenetes is based on the same VM infrastructure. Thank you for the help