CI/CD Test Container intermittently not starting up or rather being stopping with no clear indication why

[joelsteventurner]

I've setup set up Testcontainers.Net to spin up a continar running the Cosmos Emulator and use this to run my integration tests (xUnit) against.

This has been great and works very consistently when running on my desktop in Visual Studio.

I have since included these tests as part of a CI/CD pipeline and the tests are failing intermittently (roughly 50% of the time I'd say).
This failure seems to be because the container is being stopped.. for some unknown reason. (sometimes...)

The failed runs seem to happen quickly e.g. within 3-5 seconds of the container starting, I see a Stop Docker container in the log messages there is no "reason" or error I can see for why the container has been stopped. Below of samples of failed and successful runs and my container configuration.

Feels like there is a screw loose but I'm not sure where, please help.

Logs on a failed run.

[testcontainers.org 00:00:32.67] Docker container 81f09cf441a6 created
[testcontainers.org 00:00:32.67] Start Docker container 81f09cf441a6
[testcontainers.org 00:00:32.98] Wait for Docker container 81f09cf441a6 to complete readiness checks
[testcontainers.org 00:00:32.99] Execute "/bin/sh -c true && (grep -i ':0*1F91' /proc/net/tcp* || nc -vz -w 1 localhost 8081 || /bin/bash -c '</dev/tcp/localhost/8081')" at Docker container 81f09cf441a6
[testcontainers.org 00:00:34.07] Execute "/bin/sh -c true && (grep -i ':0*1F91' /proc/net/tcp* || nc -vz -w 1 localhost 8081 || /bin/bash -c '</dev/tcp/localhost/8081')" at Docker container 81f09cf441a6
[testcontainers.org 00:00:35.15] Execute "/bin/sh -c true && (grep -i ':0*1F91' /proc/net/tcp* || nc -vz -w 1 localhost 8081 || /bin/bash -c '</dev/tcp/localhost/8081')" at Docker container 81f09cf441a6
[testcontainers.org 00:00:36.23] Execute "/bin/sh -c true && (grep -i ':0*1F91' /proc/net/tcp* || nc -vz -w 1 localhost 8081 || /bin/bash -c '</dev/tcp/localhost/8081')" at Docker container 81f09cf441a6
[testcontainers.org 00:00:36.27] Stop Docker container 81f09cf441a6

Logs on a successful run.

[testcontainers.org 00:00:35.32] Docker container e8a7b31e5050 created
[testcontainers.org 00:00:35.32] Start Docker container e8a7b31e5050
[testcontainers.org 00:00:35.67] Wait for Docker container e8a7b31e5050 to complete readiness checks
[testcontainers.org 00:00:35.68] Execute "/bin/sh -c true && (grep -i ':0*1F91' /proc/net/tcp* || nc -vz -w 1 localhost 8081 || /bin/bash -c '</dev/tcp/localhost/8081')" at Docker container e8a7b31e5050
[testcontainers.org 00:00:36.77] Execute "/bin/sh -c true && (grep -i ':0*1F91' /proc/net/tcp* || nc -vz -w 1 localhost 8081 || /bin/bash -c '</dev/tcp/localhost/8081')" at Docker container e8a7b31e5050
...
[testcontainers.org 00:00:44.70] Execute "/bin/sh -c true && (grep -i ':0*1F91' /proc/net/tcp* || nc -vz -w 1 localhost 8081 || /bin/bash -c '</dev/tcp/localhost/8081')" at Docker container e8a7b31e5050
[testcontainers.org 00:00:45.86] Execute "/bin/sh -c true && (grep -i ':0*1F91' /proc/net/tcp* || nc -vz -w 1 localhost 8081 || /bin/bash -c '</dev/tcp/localhost/8081')" at Docker container e8a7b31e5050
[testcontainers.org 00:00:45.97] Docker container e8a7b31e5050 ready

Container configuration

 _dbContainer = new CosmosDbBuilder()
                          .WithImage("mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator")
                          .WithName("azure-cosmos-emulator")
                          .WithExposedPort(8081)
                            .WithExposedPort(10251)
                            .WithExposedPort(10252)
                            .WithExposedPort(10253)
                            .WithExposedPort(10254)
                          .WithPortBinding(8081, true)
                          .WithPortBinding(10251, true)
                          .WithPortBinding(10252, true)
                          .WithPortBinding(10253, true)
                          .WithPortBinding(10254, true)
                          .WithEnvironment("AZURE_COSMOS_EMULATOR_PARTITION_COUNT", "1")
                          .WithEnvironment("AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE", "127.0.0.1")
                          .WithEnvironment("AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE", "false")
                          .WithWaitStrategy(Wait.ForUnixContainer()
                            .UntilPortIsAvailable(8081))
                          .Build();

[HofmeisterAn]

I assume you are using MS-hosted agents/runners (Azure DevOps or GitHub). Unfortunately, this is a known issue. The image does not run properly on those agents (VMs). Here (Azure/azure-cosmos-db-emulator-docker#45.) is the upstream issue. Perhaps you can share your experience too, and Microsoft finally decides to fix it 😄. That is the reason why our Cosmos DB emulator tests are disabled:

testcontainers-dotnet/tests/Testcontainers.CosmosDb.Tests/CosmosDbContainerTest.cs

Line 17 in f9afd0c

[Fact(Skip = "The Cosmos DB Linux Emulator Docker image does not run on Microsoft's CI environment (GitHub, Azure DevOps).")] // https://github.com/Azure/azure-cosmos-db-emulator-docker/issues/45.

BTW: Usually, there is no reason to override the module's default configuration. I would like to emphasize that the wait strategy does not look right. We recently changed it in the module as well because it indicated readiness too early, and SSL was not ready at that point.

[joelsteventurner]

Thanks for the fast response. I found this configuration in Add support for CosmosDB Emulator and Storage Emulator (Azurite).

I think I ran into the SSL was not ready issue and was having inconsistent results on my desktop, this wait for port strategy seemed to make it work consistently. @HofmeisterAn ,where can I find documentation / examples of how to correct configure this?

[HofmeisterAn]

where can I find documentation / examples of how to correct configure this?

Usually, the tests are a good start. I am not a Cosmos DB emulator expert, but I think the module configuration should work fine. Until the next version is released, you should use the patched wait strategy. For the Cosmos DB emulator, it is important to use the provided HTTP client. The configured HTTP client will redirect the traffic to the container. I hope that helps. Maybe there are other Cosmos DB emulator experts in the community who have more insights — sorry I cannot help more.