Blocking graphql api using quarkus only executes 2 requests concurrently.

[Hardik-Parikh]

Hello all,
I have created a graphql api that just executes a cypher query in neo4j(using quarkus-neo4j extension) in a blocking way. But upon testing the graphql API with jmeter I found that throughput was equivalent to just 4 req/min.

@Query
@Blocking // has no effect
public List<Fruit> getFruitBlocking(int sleepTime) {
    logger.error(Thread.currentThread());
    Session session = driver.session();
    Result result = session.run("CALL apoc.util.sleep(%s) MATCH (f:Fruit) RETURN f ORDER BY f.name".formatted(sleepTime));
    return result.list().stream().map(record -> Fruit.from(record.get("f").asNode())).collect(Collectors.toList());
}

All requests are executed in separate worker thread. But instead of utilizing 20 threads available from the worker thread pool it only uses 2 threads. No other requests were being executed by the quarkus application when tested.

Below is the screenshot of the jmeter test:

It was also observed that after some time the server returned 5xx error:

You can find the source code of the repository here: https://github.com/Hardik-Parikh/neo4j-async/tree/main

Jmeter test plan: Fruits.zip

Zulip chat link: https://quarkusio.zulipchat.com/#narrow/stream/187030-users/topic/Blocking.20graphql.20only.20executes.202.20requests.20concurrently

[phillip-kruger]

Thanks for this ! @jmartisk @mskacelik can you have a look ?

[jmartisk]

I'll try to have a look after sorting my post-vacation backlog

[jmartisk]

I'm trying this and I can confirm that I can only execute 2 requests, but only in Dev mode. Interestingly, in dev mode the processing seems to run on vert.x-eventloop-thread-X threads, and in prod mode, on executor-thread-X threads (I don't really know why), and it uses more than two threads if necessary. Can you confirm this? Were you running dev mode for the test?

[Hardik-Parikh]

Yes, the said behavior was observed in dev mode. Also, when running in prod mode it only executes 8 requests parallelly.

[jmartisk]

Hmm. I see 12 in prod mode. But right now I have no idea where that number is coming from and why it isn't using more threads

[Hardik-Parikh]

Greetings @phillip-kruger and @jmartisk ,

I would like to inquire about any recent developments regarding this issue. It has come to our attention that this matter is impeding our progress, as we have implemented multiple GraphQL APIs with a blocking paradigm due to its current underperformance. I am interested in ascertaining whether there exist any potential workarounds for this issue. Furthermore, I would appreciate information on whether there are any forthcoming plans to address and resolve this matter in the near future.

[jmartisk]

I tried investigating it but don't really know.
The bottleneck seems to be the call to io.vertx.core.Context.executeBlocking here: https://github.com/quarkusio/quarkus/blob/3.3.0.CR1/extensions/smallrye-graphql/runtime/src/main/java/io/quarkus/smallrye/graphql/runtime/spi/datafetcher/BlockingHelper.java#L28
Here we switch from an event loop to a worker thread, but for some reason it's only using 2 threads in dev mode and 12 in prod mode, even though I set quarkus.thread-pool.core-threads to more and verified that those threads are really running and available, but unused. Where are these numbers (2 and 12) coming from?
Who could help us figure out why? @jponge and @cescoffier are on holidays.. And I'm leaving tomorrow too. Maybe @geoand has an idea

[geoand]

I'm on PTO so don't expect any feedback from me any time soon 🙂

[jmartisk]

Oh, right, everybody is out :) enjoy

[geoand]

🙏

[geoand]

How can I reproduce the behavior mentioned here?

[Hardik-Parikh]

Please clone the https://github.com/Hardik-Parikh/neo4j-async/tree/main and run either in dev mode or prod mode.
Use the Jmeter test plan: Fruits.zip to test the said behavior.

[geoand]

I tried the sample (in prod mode) with a different load tool and Quarkus behaves as expected - i.e. the worker pool expands to create new threads that are used to handle the incoming requests.

[jmartisk]

You get more than 12 threads, meaning more than 12 long-running GraphQL requests being handled concurrently?

[debu999]

Is there any findings on this we also saw blocking behaviour in our app due to which We have to move from GraphQL to Rest for a couple of the calls. Please let us know.