Performance of queries that return large number of metrics

[JarleB]

Lately I've been building some custom graphs that fetches a larger than usual (for me anyway) subset of metrics in our graphite cluster. I'm observing huge difference in time to retrieve about 1500 metrics if I go through the frontend graphite server, or if I go directly on the backend graphite server that I know the same data is located.

An example to illustrate:

jb@watershed ~ $ /usr/bin/time --format=%E curl -s 'http://frontend-server.uio.no/render/?target=collectd.*.ping.ping.*.uio.no.ping&format=raw&from=-2min' |wc -l
0:53.25
1678
jb@watershed ~ $

jb@watershed ~ $ /usr/bin/time --format=%E curl -s 'http://backend-server.uio.no/render/?target=collectd.*.ping.ping.*.uio.no.ping&format=raw&from=-2min' |wc -l
0:04.90
1678
jb@watershed ~ $

What happens is the following:

When the query is sent through the front end it hits the metric cache (same query was run just before, so cache is containing all metric locations) and instantly starts asking for single metrics from the back end server and when the backend has served up all 1678 requests, the front end returns the data to the client.

When querying the back end directly all work is done locally, which takes considerably shorter time, and replies with the same data in less than 1/10 of the time.

I wonder what would be the better approach to handle this? I'm thinking that a better approach would be to pass the unmodified urls with regexes to all backends, and collect and merge the returned data..

[deniszh]

Hello @JarleB,
Problem here lies not in size of metrics, but in metrics itself - you are using globs in your metrics and rendering of metrics with globs is not really optimized in current graphite.
Which version are you using? You can try latest 0.9.x branch of graphite-web which contains @bmhatfield's patch (#1010) for bulk glob rendering. You also can try to test parallel improvement of this patch in #1026 or even try to use @datacratic branch - #1045
For me #1010 + #1026 works quite fine.

[JarleB]

Thanks for the reply @deniszh. My wording was perhaps not correct, but yes I'm aware that the issue becomes evident because of the number of metrics returned by the glob and not the size of each metric. Seems #1010 is addressing this in particular. Will check it out.

[deniszh]

#1010 is already merged into 0.9.x btw

10:02, пн, 22.12.2014, JarleB [email protected]:

Thanks for the reply @deniszh https://github.com/deniszh. My wording
was perhaps not correct, but yes I'm aware that the issue becomes evident
because of the number of metrics returned by the glob and not the size of
each metric. Seems #1010
#1010 is
addressing this in particular. Will check it out.

—
Reply to this email directly or view it on GitHub
#1058 (comment)
.

[obfuscurity]

Gonna mark this closed. @JarleB if you try 0.9.x and find that it doesn't address your issues, please reopen. Thanks!

[JarleB]

For the record, I can can confirm that 0.9.13-pre1 (includes #1010 + #1026 ) completely eliminates this described slow behavior.

[obfuscurity]

@JarleB Awesome, thanks for the update.

[toni-moreno]

Hi @obfuscurity. I'm working with current master branch.

Are these two parches medged on the current master branch ?

[bmhatfield]

@toni-moreno - not yet. Master is sufficiently different that simply cross-merging won't do it, they need to be rewritten :-(