How to keep old metrics data

[yangtaoran]

I have this sense:

1. The first time, I push some metrics to pushgateway like bellow:
   test_metric {code=200} 2 test_metric {code=300} 3
2. The second time, I push metrics:
   test_metric {code=400} 4
   Finally, I scrap metrics from pushgateway, the result is this:
   test_metric {code=400} 4
   But what I expect is like this:
   test_metric {code=200} 2 test_metric {code=300} 3 test_metric {code=400} 4
   Is there any way to to achieve what I want?
   And two times have same job.

[beorn7]

Yes, this is the intended behavior. If you push metrics using the PUT method, all metrics in the group are replaced by the pushed metrics. If you use the POST method, only the metrics with the same name are replaced, but that still results in the behavior you have observed.

If you want to keep all the metrics above, you have to make the code label part of the grouping key. But most likely, you won't be happy with the result, either. The Pushgateway semantics is not meant to stitch together the whole metric set. Maybe you can find a way to push all the relevant metrics in one go.

[vuducmanh11]

What do you mean you won't be happy with the result. I write some code but it throw errors 400 Bad request because of the below logs

ts=2022-12-15T09:27:15.913Z caller=push.go:146 level=error msg="pushed metrics are invalid or inconsistent with existing metrics" method=POST source=[::1]:59842 err="collected metric \"test_metric_c\" { label:<name:\"instance\" value:\"\" > label:<name:\"job\" value:\"test\" > label:<name:\"project\" value:\"prjb\" > gauge:<value:1.6710964359112036e+09 > } was collected before with the same name and label values"

[beorn7]

This message means that the totality of metrics pushed to the PGW cannot be exposed for scraping in a legal way. You have probably managed to create grouping keys in a way that you can push to different groups such that you end up with the same metric in different groups. That creates a collision and is therefore invalid.