Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrate with FastHTTP (massive performance gains) #486

Open
blakewatters opened this issue Feb 25, 2020 · 13 comments · May be fixed by #500
Open

Integrate with FastHTTP (massive performance gains) #486

blakewatters opened this issue Feb 25, 2020 · 13 comments · May be fixed by #500

Comments

@blakewatters
Copy link

Proposal

We have been testing a number of load testing tools. One of my colleagues was able to generate significantly higher levels of load with fasthttploader than I was with Vegeta. After spending a bit of time drilling in, I figured out that the difference was entirely attributable to the use of the FastHTTP library.

So I decided to do a quick and dirty port Vegeta to FastHTTP. The performance impacts are staggering:

$ echo "GET http://172.31.35.140" | go run . -cpus 16 attack -max-body 0b -workers 1000 -max-workers 1500 -max-connections 2500 -rate 0 -duration 2m | vegeta report -every 3s
Requests      [total, rate, throughput]         7692795, 255634.38, 255633.41
Duration      [total, attack, wait]             30.093s, 30.093s, 114.507µs
Latencies     [min, mean, 50, 90, 95, 99, max]  61.17µs, 594.592µs, 209.441µs, 705.715µs, 1.175ms, 4.642ms, 1.036s
Bytes In      [total, mean]                     0, 0.00
Bytes Out     [total, mean]                     0, 0.00
Success       [ratio]                           100.00%
Status Codes  [code:count]                      200:7692795
Error Set:

I couldnt output a load above ~60k req/s on the stock net.HTTP implementation. I totally hacked the shit out of the codebase to get it running and there are trade-offs (loss of HTTP/2 in particular) but I pulled this off in an afternoon so I'd imagine we could land this in a few days worth of work.

There's probably an adapter approach to bifurcate the backing client. The biggest amount of work was around the wildly divergent Header implementations.

But these performance numbers are just nuts and I've verified the load from the server-side. It just absolutely cooks
@tsenart
Copy link
Owner

tsenart commented Mar 1, 2020

I played with this before and the reason I haven't pursued it further is that it'd break backwards compatibility at the library and CLI level (i.e. lack of HTTP2 support).

If you're trying to benchmark (max load) rather than load test (how does this perform at this load point), then vegeta may not be the ideal tool. I'd recommend something like wrk2, or any other tool that can achieve those high request rates.

On the other hand, yeah, it'd be nice if vegeta could yield higher request rates. But I don't think breaking backwards compatibility is worth it.

@AndreiPashkin
Copy link

@tsenart, it's possible to make an abstraction over HTTP client and use fasthttp library for HTTP and another client for HTTP2.

@tsenart
Copy link
Owner

tsenart commented Mar 3, 2020

A lot of the functional options for the Attacker would have to incur backwards incompatible changes: https://github.com/tsenart/vegeta/blob/master/lib/attack.go

@AndreiPashkin
Copy link

@tsenart, for example?

@tsenart
Copy link
Owner

tsenart commented Mar 4, 2020

@AndreiPashkin: https://sourcegraph.com/search?q=repo:%5Egithub%5C.com/tsenart/vegeta%24+file:%5Elib/attack%5C.go%24+http.&patternType=literal

@AndreiPashkin
Copy link

AndreiPashkin commented Mar 4, 2020 •
edited
Loading

@tsenart, I get it, my proposal is to make an abstraction layer over HTTP client and use fast library for HTTP1 and the standard one for HTTP/2. FastHTTP has HTTP/2 in roadmap btw.

@tsenart
Copy link
Owner

tsenart commented Mar 4, 2020

Yes, I understand, but that would be a non backwards compatible change. Unless you think of a way to maintain backwards compatibility while introducing that abstraction layer.

@AndreiPashkin
Copy link

@tsenart, hmm, you mean backwards compatibility of the API of attack.go module or backwards-compatibility of Vegeta's CLI?

@tsenart
Copy link
Owner

tsenart commented Mar 4, 2020

hmm, you mean backwards compatibility of the API of attack.go module or backwards-compatibility of Vegeta's CLI?

Of the API surface of the library. Vegeta is not only a CLI but a library.

@AndreiPashkin
Copy link

@tsenart, hmmm, perhaps we could create a wrapper over FastHTTP which would implement net.HTTP interface?

@tsenart
Copy link
Owner

tsenart commented Mar 5, 2020

I'm not sure. My intuition is that would be very hard to do.

@AndreiPashkin
Copy link

AndreiPashkin commented Mar 6, 2020 •
edited
Loading

@tsenart, Vegeta is much easier to use and more lightweight than JMeter, it already saved me a lot of time. And unlike wrk/wrk2/siege it is actively maintained. It is a very appealing idea to me to make it able to produce really huge loads.

@tsenart tsenart linked a pull request Mar 7, 2020 that will close this issue
Draft
@tsenart
Copy link
Owner

tsenart commented Mar 7, 2020

@AndreiPashkin, @blakewatters: PTAL #500, let's try to solve the open issues together.

@bbros-dev bbros-dev mentioned this issue Apr 22, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Integrate with fasthttp tsenart/vegeta
3 participants
@blakewatters @tsenart @AndreiPashkin