Fix: pool acquire allowing concurrently make new wire instead of waiting in line

[liuzhaohui]

If there is no free wire in the pool, as len(p.list) == 0, it blocks the whole pool until make wire returns.
The process of making new wire usually take several millionseconds under my environment. But if serveral tasks acquire new wires in that short period concurrently, some tasks may need to wait tens of millionseconds which brings down performance dramatically.

[rueian]

Hi @liuzhaohui, thank you. The change seems good to me but tests keep failing multiple times. There might be something we missed.

[liuzhaohui]

Hi @liuzhaohui, thank you. The change seems good to me but tests keep failing multiple times. There might be something we missed.

It seems that the testing machine lacks of enough resources when running Integration tests concurrently.
On the pull request branch, I tested the failed cases above individually, and all passed on my macbook.
And I deployed a centos with 16C32G, and ran the test command in the build.yaml and got all passed.

[rueian]

I think that was the largest machine we could have on the CI. I tried to reduce the defulat pool size d0adc18 but tests were still failing.

[rueian]

Hi @liuzhaohui, the tests have passed, which is cool! But can we keep another test on a relatively smaller scale for multiple 1ms timeouts? The original tests tried to ensure that the client could still work after timeouts. I still want to keep that but it can be done on a smaller scale.

[liuzhaohui]

Hi @liuzhaohui, the tests have passed, which is cool! But can we keep another test on a relatively smaller scale for multiple 1ms timeouts? The original tests tried to ensure that the client could still work after timeouts. I still want to keep that but it can be done on a smaller scale.

Sure, for smaller scale is not a big issue.

------- here is some explaination of the 10ms timeout chage --------
Setting BlockingPoolSize option to 10 has no effect on the testcases, as pool.size usually stays at 8 which is in line with the parrallism of 8.

On my centos, I constrained the docker server cpu/memory resource to 5c6g and I reproduced the issue. I found that the testcase with para of 'keys*100' stucked, repeatedly creating and releasing connections which took very long time. And the reason is in that testcase the context timeout is set to 1ms and there are lots of occasions the underline processing time surpass that amount of time, causing the connection to be release on context deadline excceeded error.

Both main branch and my revision has that issue, but my revision creates connection more aggressively which makes things worse under this extreme timeout setting. When I increase the timeout setting from 1ms to 10ms, the issue disappeared and it seems that the total time of the testcases decreases from 28m of main branch to 20m of my revision.

[rueian]

Merged. Thank you so much @liuzhaohui!