"Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host."

[ArchanaNeog]

Description

When concurrent TCP clients try to connect to the super socket server, a few connections fail with the error "Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host."

I have a concurrency test(MsTest)
in which 10 TCP clients connect to the supersocket server and continue sending data for 1 minute.
While running this test, I always see a few connections are closed and unable to write to the stream. However, there is no error logged in the server console.
Also for every new connection superscoket logs the message "A new session connected..." as shown below.
info: SuperSocketService[0]
A new session connected: 63ec0218-7af5-4ea4-95e2-41adcf571004
In this scenario, the new sessions connected are less than 10 connections.

Code Sample

--Csharp
SuperSockerDemo.zip contains the sample code to reproduce the issue.
UnitTest1 - It has the concurrency test which creates 10 TCP clients and connects to the server.
Once the connection is established it keeps sending data to the server for 1 minute
Program.cs has code to host the server and SupersocketRequestProcessor.cs processes the message.
The same test works as expected with other TCP server implemented without super socket.
SuperSockerDemo.zip

.NET Version

.NET 8.0

Additional Information

• Operating System: Windows 10 Enterprise

[kerryjiang]

Did you use the same client test code for other TCP server implementation?

I noticed it might be the problem of your client test code.

CancellationToken should not be passed to the methods SendAsync/FlushAsync and ReceiveAsync. Because the token is triggered by timer when send/receive is happening. An task cancelled exception will be thrown in that case and the connection will drop from client side.

I also got failures in this unit test when I pass in cancellation token but it works fine after I stopped passing the cancellation token.
https://github.com/kerryjiang/SuperSocket/blob/master/test/SuperSocket.Tests/PerfTest.cs

[ArchanaNeog]

Did you use the same client test code for other TCP server implementation?

I noticed it might be the problem of your client test code.

CancellationToken should not be passed to the methods SendAsync/FlushAsync and ReceiveAsync. Because the token is triggered by timer when send/receive is happening. An task cancelled exception will be thrown in that case and the connection will drop from client side.

I also got failures in this unit test when I pass in cancellation token but it works fine after I stopped passing the cancellation token. https://github.com/kerryjiang/SuperSocket/blob/master/test/SuperSocket.Tests/PerfTest.cs

The same client test code is tested for other TCP server implementation and works as expected.

The CancellationToken is triggered only when the DurationMinutes is expired as shown below.

           // Wait for the specified duration
            await Task.Delay(TimeSpan.FromMinutes(DurationMinutes), token);

            // Cancel the test
            cts.Cancel();

The test code cancels the token after 1 minute, but the exception is thrown before cts.Cancel(); is triggered.
The output below shows the failure occurred before the token was canceled.

[2024-12-09T13:15:02.5121029+05:30]:Starting test...
[2024-12-09T13:15:02.5478178+05:30]:An error occurred: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host... Details : at TestProject1.UnitTest1.SendMessagesAsync(CancellationToken token) in D:\WorkSpace\DicomStore-Curie\Douments\HL7ListnerForwarder\Codebase\SuperSockerDemo\TestProject1\UnitTest1.cs:line 51
[2024-12-09T13:15:02.5478206+05:30]:An error occurred: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host... Details : at TestProject1.UnitTest1.SendMessagesAsync(CancellationToken token) in D:\WorkSpace\DicomStore-Curie\Douments\HL7ListnerForwarder\Codebase\SuperSockerDemo\TestProject1\UnitTest1.cs:line 51
[2024-12-09T13:15:02.5478154+05:30]:An error occurred: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host... Details : at TestProject1.UnitTest1.SendMessagesAsync(CancellationToken token) in D:\WorkSpace\DicomStore-Curie\Douments\HL7ListnerForwarder\Codebase\SuperSockerDemo\TestProject1\UnitTest1.cs:line 51
[2024-12-09T13:16:02.5430367+05:30]:Cancelling the test...
[2024-12-09T13:16:02.5436439+05:30]:Test completed.

Added a few Console.Writeline to the test code to print the test start, cancel, and complete time.

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System.Net.Sockets;
using System.Text;

namespace TestProject1
{
    [TestClass]
    public class UnitTest1 
    {
        private const int ConcurrentConnections = 10;
        private const int DurationMinutes = 1;

        [TestMethod]
        public async Task RunTestAsync()
        {
            Console.WriteLine($"[{DateTime.Now.ToString("O")}]:Starting test...");
            CancellationTokenSource cts = new CancellationTokenSource();
            CancellationToken token = cts.Token;

            // Create concurrent connections
            Task[] tasks = new Task[ConcurrentConnections];
            for (int i = 0; i < ConcurrentConnections; i++)
            {
                tasks[i] = SendMessagesAsync(cts.Token);
            }

            // Wait for the specified duration
            await Task.Delay(TimeSpan.FromMinutes(DurationMinutes), token);

            // Cancel the test
            Console.WriteLine($"[{DateTime.Now.ToString("O")}]:Cancelling the test...");
            cts.Cancel();

            // Wait for all tasks to complete
            await Task.WhenAll(tasks);
            Console.WriteLine($"[{DateTime.Now.ToString("O")}]:Test completed.");
        }

        private async Task SendMessagesAsync(CancellationToken token)
        {
            try
            {
                using (TcpClient client = new TcpClient())
                {
                    await client.ConnectAsync("localhost", 2575); // Change the host and port as needed
                    NetworkStream stream = client.GetStream();
                    byte[] messageBytes = Encoding.ASCII.GetBytes(GenerateSampleMessage());

                    while (!token.IsCancellationRequested)
                    {
                        await stream.WriteAsync(messageBytes, 0, messageBytes.Length, token);
                        await stream.FlushAsync(token);

                        byte[] buffer = new byte[messageBytes.Length];
                        int bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length, token);
                        string result = Encoding.ASCII.GetString(buffer, 0, bytesRead);
                        //Console.WriteLine($"Received message: {result}");
                    }
                }
            }
            catch (OperationCanceledException)
            {
                // Test was canceled, ignore the exception
            }
            catch (Exception ex)
            {
                Console.WriteLine($"[{DateTime.Now.ToString("O")}]:An error occurred: {ex.Message}. Details : {ex.StackTrace}");
            }
        }

        private string GenerateSampleMessage()
        {
            StringBuilder sb = new StringBuilder();
            sb.Insert(0, '\v');
            sb.Append("Sample message");
            sb.Append('\u001c');
            sb.Append('\r');
            return sb.ToString();
        }
    }
}

Also, this failure is sporadic.

[kerryjiang]

But I cannot reproduce this issue with this unit test:

SuperSocket/test/SuperSocket.Tests/PerfTest.cs

Lines 22 to 58 in 934acd7

	[Theory]
	[InlineData(typeof(RegularHostConfigurator))]
	[InlineData(typeof(SecureHostConfigurator))]
	[InlineData(typeof(KestralConnectionHostConfigurator))]
	public async Task ConcurrentSendReceive(Type hostConfiguratorType)
	{
	var hostConfigurator = CreateObject<IHostConfigurator>(hostConfiguratorType);
	var connectionCount = 10;
	var runTime = TimeSpan.FromMinutes(1);
	using (var server = CreateSocketServerBuilder<TextPackageInfo, LinePipelineFilter>(hostConfigurator)
	.UsePackageHandler(async (s, p) =>
	{
	await s.SendAsync(Utf8Encoding.GetBytes(p.Text + "\r\n"));
	}).BuildAsServer())
	{
	Assert.Equal("TestServer", server.Name);
	Assert.True(await server.StartAsync());
	OutputHelper.WriteLine("Started.");
	using var cancellationTokenSource = new CancellationTokenSource();
	var tasks = Enumerable
	.Range(0, connectionCount)
	.Select(_ => RunConnectionAsync(hostConfigurator, cancellationTokenSource.Token))
	.ToArray();
	cancellationTokenSource.CancelAfter(runTime);
	await Task.Delay(runTime);
	var rounds = await Task.WhenAll(tasks).WaitAsync(TimeSpan.FromSeconds(10));
	OutputHelper.WriteLine($"Total rounds: {rounds.Sum()}.");
	}
	}

[ArchanaNeog]

SuperSocket/test/SuperSocket.Tests/PerfTest.cs
This perfTest uses a socket instance to connect to the server. The problem is noticed when using a TcpClient to connect to the server. If I use Socket instead of TcpClient in my test code, it works as expected.

In a production environment, we won't have any control over how the client application initiates a connection to the server.
The client application might use Socket or TcpClient to connect to the server as described in the below picture.

As a server, the expected behavior is, that both Socket-based or TcpClient-based clients should be able to connect to the server.

There is one more observation, the failure is noticed when using the asynchronous connect API of TcpClient
ConnectAsync(string host, int port) to connect to the server.
If we use synchronous connect API of the same TcpClient - Connect(string hostname, int port), it works without any issue.

[madhub]

I have also observed the same behavior when I use TcpClient connected to server written using Supersocket
Note TcpClient is wrapper over socket

[kerryjiang]

Ok, let me test with TcpClient.

[kerryjiang]

The source code of TcpClient.ConnectAsync:
https://github.com/dotnet/runtime/blob/3aa1ec5bb1f50f0a1bed9cfcac8734f742bcf24b/src/libraries/System.Net.Sockets/src/System/Net/Sockets/TCPClient.cs#L184-L189

[madhub]

The source code of TcpClient.ConnectAsync: https://github.com/dotnet/runtime/blob/3aa1ec5bb1f50f0a1bed9cfcac8734f742bcf24b/src/libraries/System.Net.Sockets/src/System/Net/Sockets/TCPClient.cs#L184-L189

Do you see any issue with implementation ?, I couldn't find any issues.
Were you able to reproduce this with TcpClient.ConnectAsync