New glob matcher report invalid values.

[kien-truong]

We're testing both 0.5.0 and 0.4.2 and getting errors with the new glob matcher.

With 0.5.0, graphite exporter error out with duplicated metrics error

collected metric "flink_assigned_partitions" { label:<name:"job" value:"ebs-dumper" > label:<name:"operator" value:"Source:-kafka:-ebs-umts" > label:<name:"subtask" value:"0" > label:<name:"task" value:"Source:-kafka:-ebs-umts-->-Process-ebs-umts-events" > gauge:<value:487941.72 > } was collected before with the same name and label values

The weird part is we don't know where the value 487941.72 comes from. We never sent that value and it didn't show up in the debug log either.

With version 0.4.2, the graphite_exporter doesn't error, but reports both valid and invalid values, which screws up the monitoring process.

flink_assigned_partitions{job="ebs-dumper",operator="Source:-kafka:-ebs-lte",subtask="2",task="Source:-kafka:-ebs-lte-->-Process-ebs-lte-events"} 5
flink_assigned_partitions{job="ebs-dumper",operator="Source:-kafka:-ebs-lte",subtask="2",task="Source:-kafka:-ebs-lte-->-Process-ebs-lte-events"} 1.80102799e+06

Reverting back to 0.3.0 or using the regex matcher will avoid this issue.

This might relate to issues #79

[matthiasr]

Could you share the mapping configuration and, if at all possible, the inputs that lead to this?

…

On Thu, May 9, 2019 at 2:56 AM Kien Truong ***@***.***> wrote: We're testing both 0.5.0 and 0.4.2 and getting errors with the new glob matcher. With 0.5.0, graphite exporter error out with duplicated metrics error collected metric "flink_assigned_partitions" { label:<name:"job" value:"ebs-dumper" > label:<name:"operator" value:"Source:-kafka:-ebs-umts" > label:<name:"subtask" value:"0" > label:<name:"task" value:"Source:-kafka:-ebs-umts-->-Process-ebs-umts-events" > gauge:<value:487941.72 > } was collected before with the same name and label values The weird part is we don't know where the value 487941.72 comes from. We never sent that value and it didn't show up in the debug log either. With version 0.4.2, the graphite_exporter doesn't error, but reports both valid and invalid values, which screws up the monitoring process. flink_assigned_partitions{job="ebs-dumper",operator="Source:-kafka:-ebs-lte",subtask="2",task="Source:-kafka:-ebs-lte-->-Process-ebs-lte-events"} 5 flink_assigned_partitions{job="ebs-dumper",operator="Source:-kafka:-ebs-lte",subtask="2",task="Source:-kafka:-ebs-lte-->-Process-ebs-lte-events"} 1.80102799e+06 Reverting back to 0.3.0 or using the regex matcher will avoid this issue. This might relate to issues #79 <#79> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#90>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABAEBWXHIN6FZQPY7IDJUTPUOHHDANCNFSM4HLWWYCQ> .

[kien-truong]

Apology for the delay. The bug is a bit hard to reproduce reliably. You might have to run issue90_test.go a couple of time to trigger it.

https://github.com/kien-truong/graphite_exporter/tree/bug-90/e2e

[PhilsLab]

+1, Facing the same issue with a big dataset.

[matthiasr]

ha, thank you, the repro case is really useful. One thing I observe is that you are sending each line individually over UDP. I'm going to try and put this into the test proper, but in some manual testing I found a data race when pushing metrics in parallel that doesn't happen when I dump them all in over one TCP connection:

==================
WARNING: DATA RACE
Write at 0x00c000190100 by goroutine 40:
  github.com/prometheus/graphite_exporter/vendor/github.com/prometheus/statsd_exporter/pkg/mapper.(*MetricMapper).GetMapping()
      /Users/mr/src/github.com/prometheus/graphite_exporter/vendor/github.com/prometheus/statsd_exporter/pkg/mapper/mapper.go:222 +0x9a9
  main.(*graphiteCollector).processLine()
      /Users/mr/src/github.com/prometheus/graphite_exporter/main.go:118 +0x1d8
  main.(*graphiteCollector).processReader()
      /Users/mr/src/github.com/prometheus/graphite_exporter/main.go:104 +0x103

Previous write at 0x00c000190100 by goroutine 39:
  github.com/prometheus/graphite_exporter/vendor/github.com/prometheus/statsd_exporter/pkg/mapper.(*MetricMapper).GetMapping()
      /Users/mr/src/github.com/prometheus/graphite_exporter/vendor/github.com/prometheus/statsd_exporter/pkg/mapper/mapper.go:222 +0x9a9
  main.(*graphiteCollector).processLine()
      /Users/mr/src/github.com/prometheus/graphite_exporter/main.go:118 +0x1d8
  main.(*graphiteCollector).processReader()
      /Users/mr/src/github.com/prometheus/graphite_exporter/main.go:104 +0x103

Goroutine 40 (running) created at:
  main.main.func2()
      /Users/mr/src/github.com/prometheus/graphite_exporter/main.go:293 +0x1db

Goroutine 39 (finished) created at:
  main.main.func2()
      /Users/mr/src/github.com/prometheus/graphite_exporter/main.go:293 +0x1db
==================

I think we're missing some serialization around accesses to mapper.GetMapping; in the statsd exporter this was already the case.

[matthiasr]

I have a potential fix in #94.