Clarify with examples for use of PUT vs POST of metrics to gateway

[daluu]

As a newcomer to prometheus, reading https://github.com/prometheus/pushgateway#put-method and https://github.com/prometheus/pushgateway#post-method, it's not clear what the end user perceived behavior is when either of the methods is called and the metrics get scraped into prometheus from the gateway.

Could you elaborate with more detail, and/or provide use case examples calling those PUT and POST endpoints showcasing the value before and after for the metrics being sent?

And/or is there any Prometheus (not push gateway) documentation that describes this PUT and POST behavior on the Prometheus side? Meaning is this replacement logic specific to push gateway, or is it a base functionality of Prometheus? If part of native Prometheus, I would like to read up more about this particular functionality (that isn't cross-referenced here).

Regarding end user perceived behavior, I'm referring to what you would/should see in Prometheus dashboard or what you would see plotting the metrics in grafana as timeseries. How does PUT vs POST affect the metric's value over time?

[daluu]

Similarly, the client libraries for push gateway give the same confusion with push vs pushAdd / push vs add, methods (naming variation based on the client library used). Once you clarify the main documentation, the client specific docs could reference the main doc about the expected behavior.

[beorn7]

I like to believe that the documentation you have linked above is concise but still complete. If someone knows a way to make them clearer without becoming much more verbose, I'll happily review a PR.

Note that it is a known issue that most of the documentation provided within the project's repositories as reference style and not very well suited for new users to actually learn how to use Prometheus. There is an effort underway to change that, so that we'll eventually get user-friendly guides with plenty of examples and explanation of context. I don't think that we can turn the current README.md into that. A more comprehensive guide should probably live in the Guides section of https://prometheus.io/ .

Final note: The examples in the Go instrumentation library are fairly comprehensive:

• https://pkg.go.dev/github.com/prometheus/[email protected]/prometheus/push#example-Pusher.Add
• https://pkg.go.dev/github.com/prometheus/[email protected]/prometheus/push#example-Pusher.Push

[beorn7]

I'll convert this to a discussion, so that people can chime in to help with explanations.

[daluu]

Are there any other ideal Prometheus (push gateway) forums to pose this discussion to? Doesn't seem we got much response here.

Based on my interpretation of the documentation, rereading it today, it sounds to me like using POST or (push) add will only update metrics that were previously sent and any other metrics in the registry/collection/set will be ignored? Use PUT or push to specifically send everything? If my interpretation is incorrect, the documentation isn't clear enough for a new user.

Also, it's not clear to me what we mean by "previously pushed" as well. Is that with respect to the push gateway tracking the metrics or the client that is pushing the metrics, tracking it? And what duration is involved with respect to previous pushes of metrics, indefinite or some expiration interval before we stop tracking what (named) metrics were previously pushed?

Some examples illustrating and comparing use of each case would be helpful to a newcomer. When do we use one over the other?

[daluu]

Ah, I see I just had to search around online, and come across related discussions. Linking here for reference:

https://groups.google.com/g/prometheus-users/c/M3xIfCczN9k
#65
https://stackoverflow.com/questions/56628091/prometheus-same-metric-with-different-labels-is-overridden-before-scraping

https://www.robustperception.io/choosing-your-pushgateway-grouping-key/

[beorn7]

On the more general point of "Are there any other ideal Prometheus (push gateway) forums to pose this discussion to?":

The forums where users discuss Prometheus problems are, sadly, very fragmented. https://prometheus.io/community/ lists quite a few, and these are just the "officially endorsed" ones (e.g. not even mentioning Stackoverflow, where there are certainly a lot of interesting nuggets to find). While fragmentation is in the nature of an open project, I would much prefer if at least the core developers could converge on one or a few channels. We had this discussion multiple times in the past, with the result that the preferences are very different (even within the core developers), and there is not a lot of appetite to converge on something like "one chat-like solution plus one forum-like solution" at the price (for most individuals) of using a solution that is not their preferred one.