Rework dashboard

[SuperQ]

Rework the dashbaord to be more useful.

• Use rate() where needed to get correct results.
• Add support for native histograms.
• Improve dashboard variables.

[SuperQ]

Fixes: #150

[SuperQ]

Fixes: #100

[SuperQ]

Fixes: #90

[dominikh]

Add support for native histograms.

This seems to be breaking the dashboard for people who aren't using native histograms. I'm getting this for the Average Latency graph:

Status: 500. Message: bad_data: invalid parameter "query": 1:1: parse error: unknown function with name "histogram_avg"

The new dashboard doesn't seem to break out multiple ping targets into their own panels anymore. This was useful to compare hosts and check if they behaved differently, e.g. due to routing. Being able to look at the sum of all hosts (by setting host and ip to all) is definitely useful, though.

And I can see how breaking them out would be bad if someone had dozens of targets. I'm not well-versed in Grafana; is there a way to add a checkbox that toggles this behavior?

[SuperQ]

What version of Prometheus do you have?

[SuperQ]

I can add the row configuration back in.

[dominikh]

What version of Prometheus do you have?

I'm on version 2.47.2. histogram_avg seems to have been added in 2.51.0, which only released in March 2024. Even then, the function is documented as

This function only acts on native histograms, which are an experimental feature.

and most users probably have their data in classic histograms, not native ones.

[SuperQ]

Yes, and that's why there is an or in the query now. If the native histogram doesn't return data, it will use the classic histogram data.

[bboehmke]

Hi,

I also checked the reworked dashboard and it looks like the or is missing in the 3rd panel Average Latency.
This results in no data for if native histograms are not enabled.

[bboehmke]

I think this should look like this

Suggested change

	"expr": "histogram_avg(rate(smokeping_response_duration_seconds{instance=\"$instance\",host=~\"$host\",ip=~\"$ip\"}[$__rate_interval]))",
	"expr": "histogram_avg(rate(smokeping_response_duration_seconds{instance=\"$instance\",host=~\"$host\",ip=~\"$ip\"}[$__rate_interval]))\nor\nrate(smokeping_response_duration_seconds_sum{instance=\"$instance\",host=~\"$host\",ip=~\"$ip\"}[$__rate_interval]) / rate(smokeping_response_duration_seconds_count{instance=\"$instance\",host=~\"$host\",ip=~\"$ip\"}[$__rate_interval])",