Cluster with consistent hashing equal to relay and carbonate

[interfan7]

For a cluster with multiple go-carbon nodes we need the same consistent hashing to be configured for relay (e.g. carbon-c-relay), carbonate and carbonapi.

I find the latter to be most difficult to grasp its configuration.
The example config doesn't seem to have a clear definition of cluster with hosts listing and a hashing config (e.g. hashing method fnv1a_ch).

The clear motivation is that for a given metric, both the relay and the carbonapi shall calculate which node "owns" this metric.

[Civil]

With carbonapi there is a problem with that - most of the requests are globbed or even contain regexes, which makes it impossible to calculate hash and know where to send requests without actually doing the request in a first place.

It was a deliberate decision to make on go-carbon side, checking if the metric is there or not as cheap as possible.

So it seems to me that support for that on carbonapi side would result in severe complications while providing little to no performance benefits.

However if you have specific use cases for that and are willing to describe them - I can change my mind (also PRs are always welcome as I tend to accept them if they don't break other people use cases).

[interfan7]

Thanks for responding.

It was a deliberate decision to make on go-carbon side, checking if the metric is there or not as cheap as possible.

What deliberate decision? To perform the globbed/regex requests on the 'go-carbon' itself?

---

My use-case is pretty much a hopeful thought that 'carbonapi' requesting the metrics from the single node which owns it would be a good resource preservation.
your explanation why it's not supported definitely makes sense to me.

What would you suggest then? To use carbonapi to transmit the requests to all the go-carbon nodes, unify the responds and send it to the metrics requester (e.g. grafana) ? Is it supported?

Does graphite-web offer something more in functionality? It seems it enables configuration of consistent hashing (CARBONLINK_HASHING_TYPE).

[Civil]

What deliberate decision? To perform the globbed/regex requests on the 'go-carbon' itself?

Yup.

My use-case is pretty much a hopeful thought that 'carbonapi' requesting the metrics from the single node which owns it would be a good resource preservation.

That is actually questionable.

But there are ways how carbonapi try to avoid always sending requests to all servers:

1. It maintain list of top level metrics (until first dot) and how they map to the server
2. If your request completes and some servers returned 0 metrics, that will be also recorded and for a while it won't send identical requests to those servers again.

Also frankly speaking, carbonapi have other tasks to do - like actual math (applying all the transofrmation functions) on a data that returned, while go-carbon is disk-bounded and have spare CPU and even RAM to expand globs (also it is virtually free for go-carbon as metrics if they are updated are cached and in that case stat call is almost free).

To use carbonapi to transmit the requests to all the go-carbon nodes, unify the responds and send it to the metrics requester (e.g. grafana) ?

Yes.

Does graphite-web offer something more in functionality?

Honestly - I don't know.

It seems it enables configuration of consistent hashing (CARBONLINK_HASHING_TYPE).

Carbon link is used only when querying the in-memory cache of carboncache (or go-carbon) - so that is useful only for data that haven't reached persistent storage yet.

And quickly looking at a code that also doesn't work for metrics with globs.

And as I've mentioned - if you think that it can be useful for your use-case - I'd like to hear more about the use-case (as I'm not convinced it will save any significant amount of resources) or you always can make a PR with that functionality implemented, as if that would be under some configuration flag (e.x. won't change default behavior) I'll gladly merge it (after it passes a review, etc, etc), no questions asked about how useful it is.

[Civil]

And to clarify a bit the thought process here:

To do glob expansion on carbonapi (same for regex matching), carbonapi need to know list of all available metrics and where to find them.
For the production setup at the time of designing, that meant that each carbonapi instance must know about at least several millions of metrics. Honestly I don't remember total amount that booking stored in graphite back then, but from some presentations that was millions of unique metrics per second and there were not that many of per-second metrics, so probably that was O(1B) unique metrics.

Roughly 100 bytes per metric name would be a good estimate, so we have 1B * 100 = 100B bytes just to store the names. And yes, you can save a bit of space by storing metrics in trie, but you need to rebuild them, etc, etc - so still that would be a lot to store and not that easy to traverse and match.

On the other hand, setup had hundreds of backends (excl. replication), so each of them stored only handful of metrics (O(1M)) and had roughly similar amount of RAM as frontend did.

That is why globbing logic was shifted to backends - basically there was not enough resources to implement it on frontend side.

And even though many years has passed since that decision was made and even though I don't work at booking anymore, I still think that in general it would not be much slower to keep that logic on backend side.

There are alternative approaches that would work and would scale, but that would be to create a separate indexing service that would resolve globbed/regex queries to list of backends but that would require rethinking some of the fundamental decisions of go-graphite stack, designing proper indexing service, protocols that backends would use to communicate with it (and what carbonapi would use), etc. That is a fun thing to do, but there are other low hanging fruits for performance here - like shifting some of the math functions to backend.

[interfan7]

@Civil thank you for such a detailed share of thought process. It was delightful to read! 🙂

When I've posted, I've not thought about the design or implementational side really.
Your elaboration seems to be very convincing.

Our bottleneck on the go-carbon node is indeed disk I/O. That's the only scale-up motivation we currently have.
Currently we have a single node which runs both go-carbon and carbonapi. Ideally I'd like to measure how much I/O is taken up by incoming metrics (write) and how much by reads (requested by carbonapi).
I'm afraid that may not be possible by system tools as the same go-carbon process is used for both write + read, and I doubt go-carob or carbonapu provide metrics which can help.
I may need to try set-up a parallel go-carbon-only dev-env node and calculate the I/O diff between the 2 nodes to yield hopefully the I/O load of the read path.

---

I've a few specific questions/comments on you last comment:

It maintain list of top level metrics (until first dot) and how they map to the server

That won't come to an assist as consistent-hashing may spread the metrics under the same top-level name to different backends.

If your request completes and some servers returned 0 metrics, that will be also recorded and for a while it won't send identical
requests to those servers again.

Is there a config to control the time or memory for that cache? Seems like a good efficiency for Grafanas' recurring requests.

that was millions of unique metrics per second and there were not that many of per-second metrics, so probably that was O(1B) unique metrics.

Would you mind to elaborate how have you calculated 1 billion unique metrics from millions? millions to a billion is a factor of 100+.
Looking at the number of Whisper files, we we have a few thousands of metrics. We may grow of course.

[Civil]

Our bottleneck on the go-carbon node is indeed disk I/O

It is known problem of go-carbon. You can try graphite-clickhouse/carbon-clickhouse setup as an alternative, but it have its own caveats, though disk i/o is way better.

Ideally I'd like to measure how much I/O is taken up by incoming metrics (write) and how much by reads (requested by carbonapi).

blkio cgroup keep track of reads and writes separately, so you can distinguish reads and writes, and you potentially can isolate ones from carbonapi by measuring a baseline without any carbonapi-related requests and then add back the load by carbonapi and just subtract one from another.

But basically go-carbon will do some amount of read I/O, but majority will be caused by carbonapi. So it won't be a huge problem to just say that all read i/o comes from carbonapi.

I may need to try set-up a parallel go-carbon-only dev-env node and calculate the I/O diff between the 2 nodes to yield hopefully the I/O load of the read path.

That is an option as well.

That won't come to an assist as consistent-hashing may spread the metrics under the same top-level name to different backends.

True.

Is there a config to control the time or memory for that cache? Seems like a good efficiency for Grafanas' recurring requests.

https://github.com/go-graphite/carbonapi/blob/main/doc/configuration.md#cache

https://github.com/go-graphite/carbonapi/blob/main/doc/configuration.md#backendcache

Are the main ones.

Would you mind to elaborate how have you calculated 1 billion unique metrics from millions? millions to a billion is a factor of 100+.

So in that case it is setup-specific. Setup had millions (>1) unique metrics per second on average, but collection time of 1s was rare, so it is safe to multiply it by at least 60 and that would give 2 orders of magnitude higher (1B) as a design goal.

Looking at the number of Whisper files, we we have a few thousands of metrics. We may grow of course.

Honestly, I wonder why you've decided to go with Graphite and especially with whisper-based setup?

Depends on what you need from monitoring, I'd either would advice to go for clickhouse-based backend, or think about prometheus or victoriametrics. There are some use-cases where whisper is still good, but its base assumption that you have a local SSD OR a lot of RAM to convert small-block random writes into sequential.

[interfan7]

But basically go-carbon will do some amount of read I/O, but majority will be caused by carbonapi.

Doesn't go-carbon perform reads for the purpose of writing too?
And I thought go-carbon does the glob/regex expansion and thus performs the disk reads to handle carbonapi requests?

Honestly, I wonder why you've decided to go with Graphite and especially with whisper-based setup?

IDK 🙂 I'm new in the team and I've a task to check how the existing Graphite node can be clusterized and become scalable.
But you've given me some great piece of info to explore and maybe broaden the possibilities I shall assess!
This infra in the company has started out about 7-8 years ago.

base assumption that you have a local SSD OR a lot of RAM to convert small-block random writes into sequential

Actually we do have a local SSD and we're good with RAM. But I feel from your words that leaning towards ClickHouse or one of the other metrics-DB infras has much more benefits than negatives compare to Whisper-based Graphite?

[Civil]

Doesn't go-carbon perform reads for the purpose of writing too?

Not as much as for purpose of reading. So there will be some, but order of magnitude of read I/O will be dominated by actual reading data to serve the users.

And I thought go-carbon does the glob/regex expansion and thus performs the disk reads to handle carbonapi requests?

Depends on configuration. If you have any kind of indexing enabled (https://github.com/go-graphite/go-carbon/blob/master/go-carbon.conf.example#L253 or https://github.com/go-graphite/go-carbon/blob/master/go-carbon.conf.example#L267), then no read requests to disk are needed as go-carbon will keep index in memory and that is enough to check if metric is there (and as well as to expand globs), however periodically it will rescan whole metric tree to update the index.

If not - then as much syscalls (stat syscall) as there are globs and metrics will be needed, but you can consider that part as partt of "reading" though.

I'm new in the team and I've a task to check how the existing Graphite node can be clusterized and become scalab

Ah... I thought that you were trying to deploy a new cluster :)

Anyway, I guess both @Felixoid and @msaf1980 can tell about their experience migrating from whisper to clickhouse. And I would strongly suggest to evaluate that option if you have opportunity/capacity to do so.

Actually we do have a local SSD and we're good with RAM. But I feel from your words that leaning towards ClickHouse or one of the other metrics-DB infras has much more benefits than negatives compare to Whisper-based Graphite?

Clickhouse have better compression support, tags support, (that is actually limitation of go-carbon), proper replication, requests for globbed data are faster, disk i/o is also lower for both reading and writing (it do way better job converting random i/o to sequential).

Drawbacks:

• you can't remove data that easily (it will be eventually performed on compaction phase)
• it is a proper database and you'll need more time to get experience with it, it will have more moving pieces (it requires 2 binaries to replace go-carbon - one on ingestion path and one on reading path)
• retention schema works differently (retention is forced on compaction phase by database or on query phase by graphite-clickhouse binary, while in whisper it is forced on ingestion - that would produce different results if you have schema like 60s:14d,1h:90d,1d:10y - first of all last portion won't be enforced by default and data will be preserved indefinitely unless you remove everything older than 10y, and moreover - aggregation function will be applied on data from the previous step - so if you've specified avg - then clickhouse will initially store raw data for 14d, then get avg point for 90d 1h schema, then when it's time to reduce resolution to daily - it will use previous result (90d one) and do avg again. Problem here is that average is non-transitive function and avg(data) != avg(avg(data/2)+avg(data/2)), but most other functions, like min. max would work just fine as they are transitive.
• small requests for a small portion of a single metrics are slower

If we are talking about VictoriaMetrics or Prometheus (they are mostly compatible on API level) - they are different databases with their own query engines (carbonapi can query both of them, but based on amount of Issues people open - either it is flawless or not a lot of people uses that in production, and I don't believe in flawless code).

On its own - it is different approach to handle metrics, tags-first, different set of functions, etc. It is de-facto industry standard way nowadays and that in my opinion is a huge benefit (I would link a story by Grammarly on how they've migrated from graphite to victoriametrics: https://www.grammarly.com/blog/engineering/monitoring-with-victoriametrics/ - I kinda know that they were using go-graphite stack before because they've contributed a lot into carbonapi prometheus backend, and as far as I know, vm supports nowadays some subset of graphite query language, but I don't know to what extent).
Drawbacks are different for both of them actually. In case of prometheus they don't have a concept of clustering, but rather use federation to achieve same thing (basically having different instances of something prometheus compatible, fetch data from other prometheus instances, etc) and on its own it doesn't have any kind of long term storage (it is single retention policy and always growing), to workaround that people uses different solutions as a Long-term storage - that can be thanos, m3db, victoriametrics, mimir, etc. (order is random), it also uses different concept of fetching data (pull rather than push).

VictoriaMetrics - implementation of the same prometheus API from scratch, it is not 100% compatible, but mostly compatible, it is faster, they try to sell some advanced features (anomaly detection, better agents, better clustering, etc - but stuff like downsampling is paid-only as well as autodiscovery and more fine-grained controls over retention policies, other things are in opensource version).

Mimir is another prometheus long-term storage implementation, but from Grafana. I haven't used it actually myself, but they claim to have support for at least high availability stuff, and maintain perfect compatibility with prometheus, but as far as I understand for long-term storage they require some cloud solution (S3, GCS, Azure Blob Storage, or anything that is compatible with one of them).

It depends on how much time you have for your project, if not too much - better to change as little as possible, but if you have time for experiments - I'd suggest to setup test clusters of at least clickhouse-based storage for graphite, and something from the prometheus world and compare how the perform.

[interfan7]

Wow you've provided so much useful info! Thanks!!

@Civil Thanks for referring others to the rescue!
Taking the effort to list strengths/weaknesses of Whisper VS Graphite and VS other metric DB infras is much appreciated and I hope it can help future readers who may bump across this thread.

My manager is leaning toward remaining in Graphite-Whisper as of now, at least for a while (business priorities, especially the prospect of getting familiarity with ClickHouse as a DB infra).

I've went over all the links. Out of curiosity - why Graphite is still actively maintained, and especially Whisper and the Python version of Graphite? Is it all for existing old users?

A specific question on carbonapi caching - as long as the cache is active and being hit, the response is not most updated right?
In our case, if we employ persistent-hashing, each metric will reside in exactly 1 node, so in that respect I'd like to have the cache long to avoid repeated request to nodes which don't even have, but doing so would be detrimental for real-time data in Grafana right? i.e. new datapoints in the nodes will not be reflected in response.
Maybe it's better that I create a separate thread for such questions, if I have such.
I'll look at the indexing option too.

[Civil]

I've went over all the links. Out of curiosity - why Graphite is still actively maintained, and especially Whisper and the Python version of Graphite? Is it all for existing old users?

It is. https://github.com/graphite-project/carbon - last commit 3 months ago

https://github.com/graphite-project/graphite-web - web interface, last commit about a month ago.

So yeah, it is. Less active than 5 years ago, but it is maintained.

A specific question on carbonapi caching - as long as the cache is active and being hit, the response is not most updated right?

So there is a cache for data itself - for that you are right, it will be served from cache, so whatever is there - you'll get, if data was updated on backend - you'll get it after cache expires. That is configurable and usually you should just use step size or something like that (there are few knobs you can fine-tune: https://github.com/go-graphite/carbonapi/blob/ea3cf74ea0cc6e3185d144626ff7b8ec9eaf3cf4/doc/configuration.md#backendcache

There is a cache about knowing where metric is https://github.com/go-graphite/carbonapi/blob/ea3cf74ea0cc6e3185d144626ff7b8ec9eaf3cf4/doc/configuration.md#expiredelaysec that knob that tunes how often it will be refreshed, but basically if response came from certain backend, carbonapi will remember that requests for that metric should be sent there and not to your other backend. That won't result in stale data, but if you manually move metric (files) from one backend or another - until cache expires requests for that metric won't be sent to another backend for same metric.

Maybe it's better that I create a separate thread for such questions, if I have such.

Up to you. Both works for me.

[interfan7]

@Civil
Thanks!! Seems like knob expireDelaySec is exactly what shall help us to avoid unnecessary queries to backend nodes which are not supposed to have those metrics.
Having said that, I'm not sure I understand what are the keys for this cache in case of globbed/regex queries?
Let's say same globbed query arrives again, with different time period and/or number of datapoints, will it be a cache hit?

but basically if response came from certain backend, carbonapi will remember that requests for that metric should be sent there and not to your other backend.

The word "but" confuses me a bit. Isn't it precisely non-zero value of expireDelaySec which enables this "carbonapi will remember that..."?

---

I expect metrics will be moved between backend nodes only when the number of backend nodes (cluster scale) changes, and that shall be a very rare scenario.

[Civil]

Having said that, I'm not sure I understand what are the keys for this cache in case of globbed/regex queries?

There are few levels of find cache:

1. Top level node name -> backends
2. Request as whole -> backends
3. Individual requests -> backends

AFAIR in that case all 3 of those will be stored in map (as key)

Let's say same globbed query arrives again, with different time period and/or number of datapoints

time period and number of datapoints will be ignored for find cache. It matters for render cache (that actually stores data).

Carbonapi doesn't take into account time and number of datapoints when making decisions where to send data (there were plans to allow people to specify backend for long term storage and for short-term, but that won't affect cache anyway, that likely will be property of the backend itself, if that would ever be implemented)

The word "but" confuses me a bit. Isn't it precisely non-zero value of expireDelaySec which enables this "carbonapi will remember that..."?

Yes, it will remember that for amount of seconds that is specified (and timer will be reset if request for same metrics will come again before it expires).