Connection has been closed by peer [Connection pool not handle load properly]

[thiagohora]

I'm executing so load tests using Mysql and R2DBC. The test is executed in different frameworks. However, only Spring boot presents the following behavior with a moderate load.

Test:

# reads a big list of zipcodes and executes a post
wrk2 -t1 -c20 -d5m -R500 -s ./post_zipcode.lua --timeout 2m -H 'Host: localhost' http://localhost:8080

Error:

rg.springframework.dao.DataAccessResourceFailureException: Failed to obtain R2DBC Connection; nested exception is dev.miku.r2dbc.mysql.client.MySqlConnectionClosedException: Connection closed
	at org.springframework.r2dbc.connection.ConnectionFactoryUtils.lambda$getConnection$0(ConnectionFactoryUtils.java:88) ~[spring-r2dbc-5.3.21.jar!/:5.3.21]
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	*__checkpoint ⇢ Handler com.myarticle.spring_project.ZipCodeApi#findById(String) [DispatcherHandler]
	*__checkpoint ⇢ HTTP GET "/zipcode/28246" [ExceptionHandlingWebHandler]
Original Stack Trace:
		at org.springframework.r2dbc.connection.ConnectionFactoryUtils.lambda$getConnection$0(ConnectionFactoryUtils.java:88) ~[spring-r2dbc-5.3.21.jar!/:5.3.21]
		at reactor.core.publisher.Mono.lambda$onErrorMap$31(Mono.java:3730) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.FluxRetry$RetrySubscriber.onError(FluxRetry.java:95) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.pool.AbstractPool$Borrower.fail(AbstractPool.java:475) ~[reactor-pool-0.2.9.jar!/:0.2.9]
		at reactor.pool.SimpleDequePool.lambda$drainLoop$9(SimpleDequePool.java:431) ~[reactor-pool-0.2.9.jar!/:0.2.9]
		at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onError(FluxDoOnEach.java:186) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:278) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:278) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:231) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.Operators.complete(Operators.java:137) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.netty.FutureMono.doSubscribe(FutureMono.java:122) ~[reactor-netty-core-1.0.20.jar!/:1.0.20]
		at reactor.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:114) ~[reactor-netty-core-1.0.20.jar!/:1.0.20]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4397) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4397) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.FluxHandle$HandleSubscriber.onError(FluxHandle.java:210) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onError(FluxPeekFuseable.java:903) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onError(MonoFlatMapMany.java:255) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.FluxHandle$HandleSubscriber.onError(FluxHandle.java:210) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onError(FluxPeekFuseable.java:903) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.EmitterProcessor.checkTerminated(EmitterProcessor.java:594) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:461) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.EmitterProcessor.tryEmitError(EmitterProcessor.java:295) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.InternalManySink.emitError(InternalManySink.java:98) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.EmitterProcessor.onError(EmitterProcessor.java:284) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at dev.miku.r2dbc.mysql.client.ReactorNettyClient.drainError(ReactorNettyClient.java:254) ~[r2dbc-mysql-0.8.2.RELEASE.jar!/:0.8.2.RELEASE]
		at dev.miku.r2dbc.mysql.client.ReactorNettyClient.handleClose(ReactorNettyClient.java:262) ~[r2dbc-mysql-0.8.2.RELEASE.jar!/:0.8.2.RELEASE]
		at dev.miku.r2dbc.mysql.client.ReactorNettyClient.access$400(ReactorNettyClient.java:53) ~[r2dbc-mysql-0.8.2.RELEASE.jar!/:0.8.2.RELEASE]
		at dev.miku.r2dbc.mysql.client.ReactorNettyClient$ResponseSubscriber.onComplete(ReactorNettyClient.java:306) ~[r2dbc-mysql-0.8.2.RELEASE.jar!/:0.8.2.RELEASE]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onComplete(Operators.java:2058) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260) ~[reactor-core-3.4.19.jar!/:3.4.19]
		at reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:468) ~[reactor-netty-core-1.0.20.jar!/:1.0.20]
		at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:260) ~[reactor-netty-core-1.0.20.jar!/:1.0.20]
		at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:404) ~[reactor-netty-core-1.0.20.jar!/:1.0.20]
		at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:419) ~[reactor-netty-core-1.0.20.jar!/:1.0.20]
		at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:473) ~[reactor-netty-core-1.0.20.jar!/:1.0.20]
		at reactor.netty.channel.ChannelOperations.onInboundClose(ChannelOperations.java:434) ~[reactor-netty-core-1.0.20.jar!/:1.0.20]
		at reactor.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:73) ~[reactor-netty-core-1.0.20.jar!/:1.0.20]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-transport-4.1.78.Final.jar!/:4.1.78.Final]
		at dev.miku.r2dbc.mysql.client.MessageDuplexCodec.channelInactive(MessageDuplexCodec.java:139) ~[r2dbc-mysql-0.8.2.RELEASE.jar!/:0.8.2.RELEASE]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-transport-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:392) ~[netty-codec-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:357) ~[netty-codec-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-transport-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) ~[netty-transport-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) ~[netty-transport-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813) ~[netty-transport-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[netty-common-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[netty-common-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:391) ~[netty-transport-classes-epoll-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.78.Final.jar!/:4.1.78.Final]
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.78.Final.jar!/:4.1.78.Final]
		at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

Dependencies:

• spring-boot-starter-parent:2.7.1
• spring-boot-starter-data-r2dbc
• spring-boot-starter-webflux
• dev.miku:r2dbc-mysql:0.8.2.RELEASE

[mp911de]

Sounds like a duplicate of r2dbc/r2dbc-pool#164. In any case, not something we can fix in Spring Data.