Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

panic in dogstatsd extended aggregation parsing #572

Closed
matthiasr opened this issue Sep 5, 2024 · 4 comments · Fixed by #579
Closed

panic in dogstatsd extended aggregation parsing #572

matthiasr opened this issue Sep 5, 2024 · 4 comments · Fixed by #579
Labels
bug

Comments

@matthiasr
Copy link
Contributor

#558 (review) – breaking this out into an issue by itself.

From @m-barthelemy:

After updating to 0.27.1, this line is triggering a panic: runtime error: index out of range [1] with length 1 for us.

Could you run the exporter with --log.level=debug and give us an example of a line that's triggering this? Ideally, add it as a test case and see if that also fails.

@GrgDev it seems that we need to check whether the splitting actually resulted in enough parts, do you have capacity to look into that?
@matthiasr matthiasr added the bug label Sep 5, 2024
@twskipper
Copy link

we got the same error, while using the 0.27.1, fixed by using 0.26.2

@m-barthelemy
Copy link

m-barthelemy commented Sep 13, 2024 •
edited
Loading

@matthiasr sorry for the delay! Here you go:

ts=2024-09-13T07:09:04.461Z caller=listener.go:97 level=debug msg="Incoming line" proto=udp line=karafka.consumer.consume.cpu_usage_second:0.12528835100000002|h|#consumer:Kafka::SharedConfigurationConsumer,topic:shared_configuration_update,partition:1,consumer_group:tc_rc_us
ts=2024-09-13T07:09:04.461Z caller=listener.go:97 level=debug msg="Incoming line" proto=udp line="karafka.consumer.consume.cpu_idle_second:  0.111090  -0.055903  -0.195390 (  2.419002)"
ts=2024-09-13T07:09:04.461Z caller=line.go:270 level=debug msg="Bad component" line="karafka.consumer.consume.cpu_idle_second:  0.111090  -0.055903  -0.195390 (  2.419002)"
ts=2024-09-13T07:09:04.461Z caller=listener.go:97 level=debug msg="Incoming line" proto=udp line=|h|#consumer:Kafka::SharedConfigurationConsumer,topic:shared_configuration_update,partition:1,consumer_group:tc_rc_us
ts=2024-09-13T07:09:04.461Z caller=line.go:124 level=debug msg="Malformed name tag" tag=consumer component=consumer
panic: runtime error: index out of range [1] with length 1

goroutine 35 [running]:
github.com/prometheus/statsd_exporter/pkg/line.(*Parser).LineToEvents(0xc000288174, {0xc0008306d7, 0x75}, {0xc00028c4b0}, {0xaf60c0, 0xc00007e540}, {0xaf60c0, 0xc0002867e0}, {0xaf60c0, 0xc0002866c0}, ...)
	/app/pkg/line/line.go:235 +0x1785
github.com/prometheus/statsd_exporter/pkg/listener.(*StatsDUDPListener).HandlePacket(0xc0002e80c0, {0xc0008300d0?, 0xc0002873e0?, 0xc0001ca7d0?})
	/app/pkg/listener/listener.go:102 +0xb3
github.com/prometheus/statsd_exporter/pkg/listener.(*StatsDUDPListener).ProcessUdpPacketQueue(...)
	/app/pkg/listener/listener.go:90
created by github.com/prometheus/statsd_exporter/pkg/listener.(*StatsDUDPListener).Listen in goroutine 67
	/app/pkg/listener/listener.go:58 +0x85

(there is no such panic triggered with 0.26.1 and older releases)

@matthiasr
Copy link
Contributor Author

Thank you! This seems like there is a linebreak in the stats line. That's, uh, tricky because we assume that this is a line-based format. Which client are you using?
As a first step, we should not panic, but it may be a bigger lift to actually support this.

@matthiasr
Copy link
Contributor Author

I'm trying to find a reference for this format but the DogStatsD datagram spec only mentions :-separated packing. I suspect something odd is going on on the sending side here, and we should refuse these lines (but we should not crash).

matthiasr added a commit that referenced this issue Sep 15, 2024
@matthiasr
Fix panic when parsing invalid lines
d6a7196 
The line

```
|h|#consumer:Kafka::SharedConfigurationConsumer,topic:shared_configuration_update,partition:1,consumer_group:tc_rc_us
```

caused a panic because the line parsing _first_ splits by `:` and then failed to
find a `|` to split on.

Check that we get at least two "line parts" (i.e. splits around `|`) when we
expect them, and if not, gracefully reject the line instead of crashing.

Fixes #572.

Signed-off-by: Matthias Rampke <[email protected]>
@matthiasr matthiasr mentioned this issue Sep 15, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix panic when parsing invalid lines prometheus/statsd_exporter
3 participants
@matthiasr @m-barthelemy @twskipper