Deadline exceed exception when getting projection list

[federicoorlandini]

Hi here,

not sure if this is the right place to raise my question. Anyway, I'm using TestContainers to spin Docker containers to run integration tests using EvenStore database.
Every integration test scenario spins a new container.
This is the test class that contains the tests:

package integration;

import com.federico.LessonBookingSystem.adapters.out.persistence.EventStoreLessonProjectionsRepository;
import com.federico.LessonBookingSystem.adapters.out.persistence.EventStoreLessonRepository;
import com.federico.LessonBookingSystem.application.usecases.CreateLessonUseCaseImpl;
import com.federico.LessonBookingSystem.application.usecases.GetLessonsOverviewUseCaseImpl;
import jdk.jshell.spi.ExecutionControl;
import Lesson.Lesson;
import Lesson.DuplicatedLessonException;
import org.assertj.core.api.Assertions;
import org.junit.jupiter.api.Test;
import org.testcontainers.junit.jupiter.Testcontainers;

import java.io.IOException;
import java.time.LocalDate;
import java.time.LocalTime;
import java.util.concurrent.ExecutionException;

@Testcontainers(disabledWithoutDocker = true)
public class GetLessonOverviewUseCaseTests extends TestContainerIntegrationTestBase {
    @Test
    public void GetLessonOverviewUseCase_createOneLesson_eventModellingScenario1() throws IOException, ExecutionException, InterruptedException, ExecutionControl.NotImplementedException, DuplicatedLessonException {
        // Arrange
        final var projectionClient = getProjectionClient();
        final var projectionRepository = new EventStoreLessonProjectionsRepository(projectionClient);

        // arranging the test scenario here......

        projectionRepository.EnsureProjectionExists();
        
        // other code here.....

        // Act
        var projection = projectionUseCase.GetLessonsProjection();

        // Assert
       // assertions here
    }

    @Test
    public void GetLessonOverviewUseCase_createTwoLessons_eventModellingScenario2() throws IOException, ExecutionException, InterruptedException, ExecutionControl.NotImplementedException, DuplicatedLessonException {
        // Arrange
        
        final var projectionClient = getProjectionClient();
        final var projectionRepository = new EventStoreLessonProjectionsRepository(projectionClient);
        projectionRepository.EnsureProjectionExists();

       // arrange the test scenario here.....

        // Act
        var projection = projectionUseCase.GetLessonsProjection();

        // asserting here
    }
}

This class inherits from this one:

import com.eventstore.dbclient.EventStoreDBClient;
import com.eventstore.dbclient.EventStoreDBClientSettings;
import com.eventstore.dbclient.EventStoreDBConnectionString;
import com.eventstore.dbclient.EventStoreDBProjectionManagementClient;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.utility.DockerImageName;

import java.util.Map;

// This is the base class for all the integration tests using the TestContainer.
// This class contains all the common code shared by all the integration tests using the TestContainer library
public abstract class TestContainerIntegrationTestBase {
    protected static final String CONNECTION_STRING_TEMPLATE = "esdb://admin:changeit@localhost:%s?tls=false&tlsVerifyCert=false&defaultDeadline=30000";
    protected static final int EVENT_STORE_PORT = 2113;

    protected final Map<String, String> ENV_CONFIGURATION = Map.ofEntries(
            Map.entry("EVENTSTORE_ALLOWUNKNOWNOPTIONS", "true"),
            Map.entry("EVENTSTORE_INSECURE", "true"),
            Map.entry("EVENTSTORE_CLUSTER_SIZE", "1"),
            Map.entry("EVENTSTORE_RUN_PROJECTIONS", "All"),
            Map.entry("EVENTSTORE_START_STANDARD_PROJECTIONS", "true"),
            Map.entry("EVENTSTORE_EXT_TCP_PORT", "1113"),
            Map.entry("EVENTSTORE_EXT_HTTP_PORT", "2113"),
            Map.entry("EVENTSTORE_ENABLE_ATOM_PUB_OVER_HTTP", "true"),
            Map.entry("EVENTSTORE_ENABLE_EXTERNAL_TCP", "true")
    );

    @Container
    protected final GenericContainer<?> eventStoreDbContainer =
            new GenericContainer<>(DockerImageName.parse("eventstore/eventstore:latest"))
                    .withExposedPorts(EVENT_STORE_PORT)
                    .withEnv(ENV_CONFIGURATION);

    @BeforeEach
    public void beforeEach() {
        eventStoreDbContainer.start();
    }

    @AfterEach
    public void afterEach() {
        eventStoreDbContainer.stop();
    }

    protected EventStoreDBProjectionManagementClient getProjectionClient() {
        var settings = getSettings();
        return EventStoreDBProjectionManagementClient.create(settings);
    }

    protected EventStoreDBClient getEventStoreDbClient() {
        var settings = getSettings();
        return EventStoreDBClient.create(settings);
    }

    private EventStoreDBClientSettings getSettings() {
        var port = eventStoreDbContainer.getMappedPort(EVENT_STORE_PORT);
        var connectionString = String.format(CONNECTION_STRING_TEMPLATE, port);
        var settings = EventStoreDBConnectionString.parseOrThrow(connectionString);
        return settings;
    }
}

And this is the ProjectionRepository class that implements the EnsureProjectExists() method:

import com.eventstore.dbclient.EventStoreDBProjectionManagementClient;
import Lesson.Lesson;
import com.federico.LessonBookingSystem.application.projections.ports.out.persistence.LessonProjectionsRepository;
import jdk.jshell.spi.ExecutionControl;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Repository;

import java.time.LocalDate;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutionException;

@Repository
public class EventStoreLessonProjectionsRepository implements LessonProjectionsRepository {
    // This class is only used to deserialize the result of the projection LessonProjection
    private static class LessonProjectionResult {
        public ArrayList<Lesson> lessons;
    }
    private final EventStoreDBProjectionManagementClient projectionClient;

    @Autowired
    public EventStoreLessonProjectionsRepository(EventStoreDBProjectionManagementClient projectionClient) {
        this.projectionClient = projectionClient;
    }

    @Override
    public void EnsureProjectionExists() throws ExecutionException, InterruptedException, ExecutionControl.NotImplementedException {
        // Must check if the projection needed by this repository is present in the event store
        if( ProjectionExists() ) {
            System.out.println(String.format("Projection %s already exists. Skipping the creation.", LESSONS_PROJECTION_NAME));
        }
        else {
            System.out.println(String.format("Projection %s doesn't exist. Creating it.", LESSONS_PROJECTION_NAME));
            CreateProjection();
            System.out.println("Projection created");
        }
    }

    .......

    private boolean ProjectionExists() throws ExecutionException, InterruptedException {
        System.out.println("Reading projections");
        var projections = projectionClient.list().get();

        System.out.println("Projections have been red");
        return projections
                .stream()
                .anyMatch(item -> item.getName().equalsIgnoreCase(LESSONS_PROJECTION_NAME));
    }

    private void CreateProjection() throws ExecutionException, InterruptedException {
        final String LESSON_STATUS_JS_SCRIPT = "fromAll()\n" +
                "    .when({\n" +
                "        $init: function () {\n" +
                "            return {\n" +
                "                lessons: []\n" +
                "            }\n" +
                "        },\n" +
                "        LessonCreated: function (state, event) {\n" +
                "            const bodyraw = JSON.parse(event.bodyRaw);\n" +
                "            state.lessons.push({id: bodyraw.lessonId,\n" +
                "                date: bodyraw.date,\n" +
                "                startTime: bodyraw.startTime,\n" +
                "                endTime: bodyraw.endTime,\n" +
                "                maxNumberAttenders: bodyraw.maxNumberAttenders,\n" +
                "                status: 'OPEN'});\n" +
                "        }\n" +
                "    })\n" +
                "    .outputState()";

        projectionClient.create("LessonProjection", LESSON_STATUS_JS_SCRIPT).get();
    }
}

When executing the Gradle build, the first call to the method projectionRepository.EnsureProjectionExists() works, it returns the list of the projections and the test pass, but when Gradle executes the other test case and it executed the second time the projectionRepository.EnsureProjectionExists() then the execution stucks and I after the timeout the test fails with the error:

java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 29.999275400s. Name resolution delay 0.000000000 seconds. [closed=[], open=[[remote_addr=localhost/127.0.0.1:63215]]] at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.federico.LessonBookingSystem.adapters.out.persistence.EventStoreLessonProjectionsRepository.ProjectionExists(EventStoreLessonProjectionsRepository.java:61) at com.federico.LessonBookingSystem.adapters.out.persistence.EventStoreLessonProjectionsRepository.EnsureProjectionExists(EventStoreLessonProjectionsRepository.java:31) at integration.EventStoreLessonProjectionsRepositoryTests.GetLessonOverviewUseCase_createOneLesson_eventModellingScenario3(EventStoreLessonProjectionsRepositoryTests.java:124)

For a reason that I'm not getting, the second call to the projectionClient.list().get() (see ProjectionExists() method in the class EventStoreLessonProjectionsRepository) stucks.

I checked the logs and I can confirm that this is the line of code where the execution stops until the timeout.

Any suggestion?

Thank you.

[w1am]

Hey @federicoorlandini

It's hard to read the code the way it's been formatted. Is there a repository we can look at to see the code?

[federicoorlandini]

Hi @w1am ,
I fixed the text formatting for the code in my initial message. I hope that it is now more readable.

Looking forward to reading your thoughts about the issue I presented.

Thank you.

[w1am]

This might be caused by lingering connections from the previous container. I would make sure to close the client before stopping the container.

Perhaps something like this:

@AfterEach
public void afterEach() {
    if (projectionClient != null) {
        projectionClient.shutdown().get();
    }
    if (eventStoreDbClient != null) {
        eventStoreDbClient.shutdown().get();
    }
    eventStoreDbContainer.stop();
}

We do the same in our tests. See ClientTracker.java

[federicoorlandini]

Hi William, thank you for your suggestion. Unfortunately, the issue persists even after adding the call to the shutdown() method as you suggested. Since the issue persists and I didn't find any other information/suggestion, I'm going to drop EventStoreDB as event store database and I'll migrate to PostgreSQL. Thank you for the help. Best regards, Federico Il giorno lun 7 ott 2024 alle ore 08:58 William Chong < ***@***.***> ha scritto:

…

This might be caused by lingering connections from the previous container. I would make sure to close the client before stopping the container. Perhaps something like this: @AfterEach public void afterEach() { if (projectionClient != null) { projectionClient.shutdown(); } if (eventStoreDbClient != null) { eventStoreDbClient.shutdown(); } eventStoreDbContainer.stop(); } — Reply to this email directly, view it on GitHub <#287 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACMSCQEFIZMWRNJWRERMBG3Z2IWJZAVCNFSM6AAAAABPDZOJ32VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJWGA3DONJVHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>