Do /metrics/find queries in parallel

[jraby]

Based on what was done to fetchData() in #1026.

/metrics/find/ queries are used less often since #1010 went in, but the composer still uses them.
This makes browsing metrics through the composer much less painful :-)

The code has been very lightly tested (it is December 30th code after all), but it works on a 10 node cluster.

cc @bmhatfield

[obfuscurity]

We've got to cut off any more commits to 0.9.x ASAP but if this results in a much better experience for the user then I'm tentatively ok with it.

[jraby]

fwiw, I still think using a threadpool makes the code easier to read:

• datacratic@14aee68
• datacratic@887c0b8

These would do mostly the same as the current PR, but with much less boiler plate code. (if we remove all the if USE_THREAD stuff)

[SEJeff]

I'd also really like to see this against master

[jraby]

As I've said in another thread a few month ago, we don't currently have an environment to test master, but we do have multiple 0.9.x clusters... So for the short term, I'm not sure I'll be able to help with forward porting this kind of patches to master.

As for the performance, with a 10 node cluster where 4 nodes are 80ms away from the cluster head, the original code takes ~400ms to return results for a /metrics/find query, whereas the new code takes ~170ms.

Regarding the commit cutoff to 0.9.x, I'm currently updating our fork to the latest code (with threads), and I planned to cleanup my prefetch cache patch from issue #1045 since, in our case, using that patch makes queries with multiple targets more than twice as fast.
Would that be of interest to the project, or is the focus shifting on master from now on? (or at least after 0.9.13 is released)

[obfuscurity]

After 0.9.13 is tagged and shipped, there should be nothing except for bugfixes going into 0.9.x.

[SEJeff]

What are your thoughts on merging this into 0.9.x @obfuscurity?

[obfuscurity]

@SEJeff I'm an idiot when it comes to threaded Python (or any threaded code, tbh) but if it looks ok to you, I'm ok with it being merged.

[deniszh]

Didn't test it on our clusters (because of vacations and production freeze), but looks perfectly legit to merge (and complement to #1026). 👍

[bmhatfield]

This also looks fine to me, though I haven't tested it. It closely follows the pattern in #1026.