Premature server response causes HTTP client to crash

[kachayev]

If a server sends response while a client is still transferring request body, the exception raised by netty (see details below) is caught in client handler and causes an appropriate connection to be "stuck" (which is another problem). Regarding RFC2616,

An HTTP/1.1 (or later) client sending a message-body SHOULD monitor the network connection for an error status while it is transmitting the request. If the client sees an error status, it SHOULD immediately cease transmitting the body. If the body is being sent using a "chunked" encoding (section 3.6), a zero length chunk and empty trailer MAY be used to prematurely mark the end of the message. If the body was preceded by a Content-Length header, the client MUST close the connection.

Server:

(defn no-ops-handler [_]
  {:status 200
   :body "OK"})

(http/start-server no-ops-handler {:port 2018
                                   :request-buffer-size 16})

Client:

(http/post "http://localhost:2018" {:body (map str (range 1e6))})

Client logs an exception:

[aleph-netty-client-event-pool-7] WARN aleph.http.client - error in HTTP client
io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest, state: 2
<< … >>
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:107)
        at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
        at io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:305)
        at io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:135)
        at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802)
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:837)
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071)
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:304)
        at aleph.netty$write_and_flush.invokeStatic(netty.clj:235)
        at aleph.netty$write_and_flush.invoke(netty.clj:232)
        at aleph.http.core$send_streaming_body$fn__7261.invoke(core.clj:267)
        at aleph.http.core$send_streaming_body.invokeStatic(core.clj:262)
        at aleph.http.core$send_streaming_body.invoke(core.clj:253)
        at aleph.http.core$eval7287$send_message__7294$fn__7295.invoke(core.clj:415)
        at aleph.http.core$eval7287$send_message__7294.invoke(core.clj:414)
        at aleph.http.client$http_connection$fn__7701$fn__7702$f__6731__auto____7706.invoke(client.clj:541)
        at aleph.http.client$http_connection$fn__7701$fn__7702$fn__7708.invoke(client.clj:540)
        at clojure.lang.AFn.run(AFn.java:22)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
        at manifold.executor$thread_factory$reify__1987$f__1988.invoke(executor.clj:47)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest, state: 2
        at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86)
        at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167)
        at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89)
        ... 31 more

[kachayev]

For the refernce. Somewhat related issue for Netty (still open) and attempt to workaround in AsyncHttpClient.

[kachayev]

After some investigation... apart from juggling with ClosedChannelException handling through the pipeline, to fix this issue, we need to switch off ChannelOption.AUTO_CLOSE: default value is true, which means that any write failure leads to closing the connection. In such a situation, we may lose any data that was available for reading already but didn't make it through our handlers. Effectively, that's exactly what causes the problem here: the server sends the response & closes the connection but we were too busy writing and flushing chunks of the data until it's too late to try to read.