Run again with 480 egines, 20 tasks/engine (top is main, 3093386f , bottom is this pr, a98a4fa). Workload is load-balanced submission of random 0-1s tasks (same seed), 20 tasks/engine for a total of 9600 tasks.

[Cathy131415]

    Run again with 480 engines, 20 tasks/engine (top is main, 3093386f , bottom is this pr, a98a4fa). Workload is load-balanced submission of random 0-1s tasks (same seed), 20 tasks/engine for a total of 9600 tasks.

Can see that while the client is working to produce the tasks, there is still contention between serializing in the main thread and actually sending in the io thread until the main thread is done (purple line). This completes 1s faster in this PR (7.6s vs 8.4s). The first result doesn't arrive for 2 more seconds, which is really around when the last real send completes and receives start being processed.

The bubble can be seen around 11s in main, which is where sends and receives are both being processed, and this is gone after this PR.

Originally posted by @minrk in #534 (comment)

[Cathy131415]

the problem is very difficult, i think it that will python and java that can

[minrk]

Looks like this issue was created by mistake, copying a comment out of PR #534. @Cathy131415 can I ask what your goal is here?