[BUG] Duplicated series with VictoriaMetrics backend #646
Comments
|
Can you describe a way how can I replicate that behavior on an artificial setup (e.x. imagine that I've just installed VM and carbonapi, what should I feed it to trigger that behavior)? As my test setup for VM + carbonapi looks like that: write path: telegraf -> (influxdb writer) -> VM read path: carbonapi -> VM With that setup I've never seen such behavior. I'll try to reproduce it though again (not sure how I can delay data merges, it might be dependent on what VM returns). But I guess in my setup I do not usually see unmerged data (I do not have that much of a load, I'll try to throttle disk i/o for vm though to replicate that behavior). It might be helpful if you can provide what VM returns on typical request from carbonapi (any metric that would trigger that behavior would work). You can enable debug logs in carbonapi and it should eventually print exact query that it do towards VictoriaMetrics and then you can query some data and replace actual metric names with something that doesn't contain any sensitive information (as well as for example replacing all data with |
|
Also are you sure that it happens for both But version detection relies on you to use official packages or to at least that VM returns version correctly on |
|
Unfortunately, I had to terminate my setup because the debug mode quickly filled up the root disk, so I can't pull the logs anymore, sorry. |
|
In the end, I decided to give it another try with a single-node setup. And the weirdest thing with this bug is that everything works fine on the single-node version. Perhaps, something's wrong with how carbonapi interacts with vmselect? |
|
And now I was able to reproduce it on the single-node version too. It appears not so often, but still. Maybe flags -selfScrapeInterval=60s and -search.disableCache play some role here because I was able to see it only when I restarted the node with these parameters. Protocol used - victoriametrics. |
|
Thx. I'll try to run VM with that flags and see how it will go. I likely will try that next week or so. I also had plans to try implement Victoria Metric's Graphite /render API (or at least remember why I haven't done that before). Maybe there it will work better (as well). |
|
FYI: unfortunately, the same problem appears even with Clickhouse as a backend. |
|
And what would be the replies from the backend in that case? As carbonapi by design do not do any deduplication of metrics that backend returned. I mean can you provide example query, reply from the backend, reply from a carbonapi. |
|
Yes, here they are: I've also compared the data in CH to what graphite-clickhouse returned, and the results seem to match (both the values and the total number of values): As you can see, the duplication happens on the carbonapi level. Maybe you have any idea what could be causing this? |
|
Can you enable log level debug in carbonapi and post a full log for the request that duplicates metrics? Ideally from the start of application (it have a dump of config in a way how it was parsed, etc). |
|
|
I've tried out different versions and different backend config types (old-fashion backend vs newer backendv2), here are the configs: v1: v2: Here are the results for the clickhouse backend: As you can see, the latest stable version available as rpm package is 0.12.6-1. Maybe that gives you a hint. By the way, rpm package uninstallation doesn't work correctly as well - it leaves a systemd service running. |
|
Can you test if the problem is still there if you have |
|
No difference, still having duplicated results. |
|
I've tried to use your config from #646 (comment) (v2 version) but I cannot reproduce metrics duplication. However I've noticed an odd thing in your logs: And to be precise that part: As that is populated here: carbonapi/zipper/broadcast/broadcast_group.go Lines 470 to 476 in bb33f26 as I'll push some changes to logging to get rid of duplicate As about your questiuon about Also about Also if backend somehow duplicate results, carbonapi won't do any deduplication on it's own. |
Log some metrics for the requests That should help to debug #646
|
In current master I've added some metrics to debug log level. I hope it will help to better understand what's going on. |
Failed requests do not populate response structs so they shouldn't be logged Related to #646
Describe the bug
Hi. Any metric queried from carbonapi results in multiple absolutely identical series with the same name. The number of duplicates seems to depend on whether there were any data merges in VictoriaMetrics (e.g. the number of duplicates can be 5, and after forcing a merge it reduces to 2).
CarbonAPI Version
carbonapi-0.15.4~1-1
Logs
Literally any metric and request should reproduce the issue, but let me know if request-response logs are still needed.
CarbonAPI Configuration:
VictoriaMetrics Configuration:
Cluster version, 1.64.1. Metrics are ingested like this: carbon-c-relay -> vmagent -> vminsert -> vmstorage, and read like this: grafana -> carbonapi -> vmselect -> vmstorage.
The text was updated successfully, but these errors were encountered: