Performance issue while moving from v3.4.2 to v3.7.3

[hlecnt]

Hi all,

I'm starting a discussion here because I'm facing performance issue while moving my app from aiohttp v3.4.2 to v3.7.3 (python 3.6.12 under a debian slim buster container).

The app is a kind of HTTP load balancer doing many things... I spotted the issue in the app. The problem comes from the time taken by the app to connect to its upstreams due to the following timeout constraints:

aiohttp.ClientTimeout(connect=0.5, sock_connect=0.5)

When testing with aiohttp 3.7.3 some connection can't established fast enough. This was not the case with aiohttp 3.4.2.
As the app does lots of things I wrote a minimalist load tester that simulate the app. It's a docker-compose app available here.

The issue seems the be reproduced when the test app is running on a GCP compute instance e2-standard-8. It has few cores but spots the issue (from my point of view).

Here are the results (quick summary):

• Best results are achieved with aiohttp 3.4.2 + uvloop, python 3.6.12
• whatever the python version (3.6.12 or 3.8.12) and the event loop used, when using aiohttp 3.7.3 the load test generates lots of failures.

Please find attached locust reports for more details.

Does anybody has already experienced this kind of issue while upgrade to a newer version ?

Best regards,

e2-standard-2-8GB.zip

[hlecnt]

Hi all,

Just a little update about the issue. Same problems are observed using Python 3.9.7 + aiohttp 3.7.3 on latest python debian slim base image.

Benchmark tool is still there:
https://github.com/hlecnt/aiohttp-load-test

Regards,

[hlecnt]

Thanks for these explanations !

gcc toolchain is already installed in the 'build' containers.

About aiohttp 3.7.3, your platform-specific wheel is downloaded:

Collecting aiohttp==3.7.3
  Downloading aiohttp-3.7.3-cp36-cp36m-manylinux2014_x86_64.whl (1.3 MB)

Import of 100% C-extensions modules in my REPL are OK too. And .so built files are present in the site-package:

/usr/local/lib/python3.6/site-packages/aiohttp/_frozenlist.cpython-36m-x86_64-linux-gnu.so
/usr/local/lib/python3.6/site-packages/aiohttp/_helpers.cpython-36m-x86_64-linux-gnu.so
/usr/local/lib/python3.6/site-packages/aiohttp/_http_parser.cpython-36m-x86_64-linux-gnu.so
/usr/local/lib/python3.6/site-packages/aiohttp/_http_writer.cpython-36m-x86_64-linux-gnu.so
/usr/local/lib/python3.6/site-packages/aiohttp/_websocket.cpython-36m-x86_64-linux-gnu.so

About aiohttp 3.4.2, .so files are also present in the site-packages directory.

=> C-extensions seems to be present and correctly used.

About Debian distro, do you any recommendation about the 'best' distro for AIOHTTP ? I have already tried Alpine but faced some weird memory usage (probably due to musl) and very long build times for some Python libraries.

[webknjaz]

I don't have specific distro recommendations. Probably whatever suits you best would work. You just need to be aware of these quirks.
With Alpine, it's a bit tricky: older aiohttp versions seamlessly fall back to pure-python unless you do something about it + there are no musllinux wheels for those. But we've just started shipping those musllinux wheels with the release of aiohttp v3.8.0 so going forward you'll get pre-compiled wheels under Alpine for as long as you have a modern enough pip (because it vendors packaging which effectively holds the list of tags like manylinux2014 or musllinux_1_1 that need to be detectable for pip to understand that a given platform is compatible with a certain wheel).
I've personally grown to like RPM-based distros during the last several years so maybe Fedora/CentOS Stream/etc would work for you too but I keep using Gentoo Linux on my laptops and don't feel any need to migrate. Although I'm biased here so take it with a grain of salt. Plus, we (the upstream) don't test anything other than what GHA provides us with by default (hence only Ubuntu).

Now that you've identified that you have C-extensions, unless you set AIOHTTP_NO_EXTENSIONS=1 in runtime, it looks like it should always use the optimized code. Oh, and you could experiment with actually setting AIOHTTP_NO_EXTENSIONS=1 to see if there's a performance drop compared to your normal invocation (just to double-check).

Now that we know that we know that it may be related to the C-extensions, you could try comparing lower versions like aiohttp 3.5/3.6, and maybe even their patch versions too, against your baseline 3.4.2. You're looking to bisect when the drop was introduced. aiohttp 3.4 is quite old, a lot has happened since then, it's hard to tell now if there was some specific change that affected the performance negatively. Besides, with 3.8.0 we've replaced the HTTP parsing library that stopped being maintained with its successor fork (http-parser -> llhttp from the NodeJS ecosystem). This doesn't affect your testing but adds to the complexity of bisect. This is why I recommend trying one minor release bump at a time with all of the other runtime setup being exactly the same. If you find two minor versions that differ significantly, you could try going through the patch releases to help make the diff smaller.

I haven't got any better ideas for now but maybe @asvetlov would be able to point at some.

[hlecnt]

I did some tests to identify the "bad" version. The drop was introduced between v3.6.3 and v3.7.0b0. I have to update my tool in order to build aiohttp from the source and try to find THE commit. I'll let you inform.

Thanks again for the time spent on this subject.

[hlecnt]

Some news,
I found the commit introducing the drop: link

It's related to timeout handling and my use case uses small timeout (ClientTimeout) values to reduce latency of some processing:

timeout = ClientTimeout(
    total=1, connect=0.5, sock_connect=0.5
)

The 'issue' appears with and without C-extentions. I've already tested increasing these values and it's fine but this may add regression in processing in case of "too high response" time. Thus I prefer to find a workaround if any available.

My understanding is that former code (3.6.3) uses ceil() for all timeout values so that the small values I use have an extra delay (hidden due to ceil()) up to 1 second (worst case). This former behaviour hides the issue seen with the new version as timeout are more precise for small values.

Thus my issue is not a performance issue: to reproduce the same behaviour with the latest version of AIOHTTP I have to increase the timeout value by 500ms, am I right ?

[webknjaz]

Sounds reasonable. I'm not very well familiar with that code. Note that the new version still applies ceil() when the timeout is greater than 5. I wonder if that constant of 5 could be moved to a more centralized variable, maybe with a possibility to tweak it for a given app...

cc @asvetlov @Dreamsorcerer

[asvetlov]

I think the reason is a little different.
After the mentioned change, your code schedules a lot of possible timeout events that pollute event loop internal structures even if timeouts never occur.
Before the change, all 'close' timeouts were grouped by scheduling at the ceiled time. Many possible scattered timeouts are grouped and called at most once per second, not 1000+ timeouts at different times during this second.
The ceiling works fine but caused problems with test environments that sometimes want to check timeout without relatively long waiting for a second. The ceiling now is processed only if timeout >= 1 sec, and disabled for tiny timeouts.

I believe you can increase timeouts to 1 sec to enable the ceiling; maybe all other parts of your system still work smoothly.

Theoretically, we can add a configured threshold for switching to the ceiling approach but I really don't want to pollute the public API with esoteric parameters that are almost never used.

[hlecnt]

Explainations are clear from my point of view, thanks!

On my side, the current status is the following: by setting the timeout value to 1 second instead of a low value of 500ms, the original reported issue disappears with latest aiohttp version (3.8.0): good news!
Nevertheless I still have too much loop wakeups compared to former aiohttp version, so there is a risk of performance issue.

Five seconds timeout is a too high value in case of a missing peer (for my use case).

Is there any way/chance to configure this value or decrease it to 1 second, in order to limit loop wakeups (as for aiohttp versions < 3.7.0) ?

[Dreamsorcerer]

Sounds reasonable to have some way to control this explicitly. Either adding an argument to always ceil(), or allow passing a when instead of a timeout in which case the user can ceil(), round() or whatever to match their particular tradeoff between performance and accuracy.

[asvetlov]

Absolute times are not practical, especially for things like "timeout between 'read' calls".

Regarding the explicit rounding control -- sounds good, I'm open to an API proposal.

[hlecnt]

Please find a first proposal as a Draft PR here.

Please note changes in requirements/test.txt and typing-extensions.txt because of pip dependency error when starting make test.

I didn't do further tests for now. By default changes did not modify current user behavior and they are limited to one file.
It's a global configuration, not a per request configuration but allow to recover the former behavior. From my point of view, using an environment variable makes sense in that case because it's a low level configuration that mostly users don't care. Nevertheless I don't kown how you manage environment variables in your project.

"changes did not modify current user behavior" is not very true because the PR updates change ceil_timeout(). This function was not modified in aiohttp 3.7 so maybe we can keep the current ceil_timeout() behavior with the hard-coded 5 seconds.

Please let me know.

[hlecnt]

@asvetlov @Dreamsorcerer Please find a new version for the configuration of the ceiling threshold:
6316

I have tested this version using my load testing tool. The library achieves better performance now: no more 100% request error rate. Nevertheless it's not able to achieve the same performance as aiohttp 3.6.3. The error rate during the load test is twice the error rate when using aiohttp 3.6.3. But it may be acceptable for my use case. I need more time to confirm and to investigate.

By default, the current behavior is kept, to restore the former ceil of timeout values, the user has to:

• configure the threshold through ClientTimeout object, for client requests
• configure the web.Application itself using already existing handler_args:

app = web.Application(handler_args={"timeout_ceil_threshold": CEIL_THRESHOLD})

Regards,