Bad rate on log to metric counter

[LinTechSo]

If I increase the replica number of vector instances on the Kubernetes cluster and I put a log to metric transformation on the pipeline that counts a specific field, the rate of that count will be Invalid...
it gives us zero rates but we are using demo logs with a time intervals of 1 second...

[jszwedko]

Hi @LinTechSo !

Apologies, I'm not quite sure I understand the issue. Could you provide the Vector configuration you are using, the expected output, and the actual output?

[LinTechSo]

Hi @jszwedko
i have some vector piplines that i use them as a log shippers
and also i deployed ds vector log generator manifests that produce logs with rate of 1 per minutes and send them to the kafka then my main vector read that from kafka and count it and send it to loki:
source[Kafka] ------> transform[log counter] ----> sink[loki]
so my main vector counter configuration:

        type: "log_to_metric"
        inputs:
          - to_json
        metrics:
          - type: counter
            name: vll_tag_total
            field: tag
            namespace: vll
            tags:
              tag: "{{tag}}"
              source_cluster: "{{cluster}}"
              source_node: "{{node_name}}"

the rate of this log to metric counter become zero in some times per node (one per 5 minutes for example) after i increased the replica number but we don't have any log loss ...
i set the vector deployment replica to 1 and everything works fine but why ?

( Kubernetes 1.21 cluster with vector version 0.22.0 )

this link is a screenshot of my grafana that shows my issue very well

the first panel is counter rate with replica 1
the seccond panel is counter rate with replica 3

[jszwedko]

Could you share the loki sink configuration? One guess I have is that you have multiple Vector instances writing to the same Loki stream, which could cause problems.

[LinTechSo]

      loki_sink:
        type: loki
        inputs:
          - filterـtest
        endpoint: http://loki.test.svc:3100
        out_of_order_action: rewrite_timestamp
        labels:
          tag: "{{tag}}"
        batch:
          max_bytes: 1024000
        buffer:
          type: memory
          max_events: 5000
          when_full: block
        tenant_id: "test"
        remove_timestamp: false
        encoding:
          codec: json

[jszwedko]

Thanks @LinTechSo . I think the issue here is that, when you have multiple replicas, they will all be writing to the same Loki stream since the tag label you have won't be unique across them. I'd recommend trying to add an additional label that is specific to the Vector instance like host: "{{host}}" that will ensure each Vector writes to its own stream.

Just another note, if you are running Loki with version 2.4 or later we recommend setting out_of_order_action: accept since Loki can handle the out-of-order writes.

[LinTechSo]

Hi @jszwedko.
Thanks for the info.
I tested it again with your recommendations and also put a print sink but the problem still persists.
I open an issue for this with more details and tests
please see #13452

[JeanMertz]

Closing this, as discussion is continuing in #13452.