Transaction can't spin off additional, unrelated transactional work via ExecutorService

[lightbody]

Expected Behavior

In Micronaut 3, I could write code that would execute in a transactional context and then submit work via an ExecutorService to happen in it's own, separate transaction. Our application makes heavy use of this pattern and it seems reasonable to expect it to work in Micronaut 4.

Instead, in Micronaut 4, it appears that because of ExecutorServiceInstrumenter the context from the initial transaction is dragged into the downstream runnable. The result is when any database work starts (including the attempt to start a new transaction), it fails with an error from Hikari that "the connection is closed".

Actual Behaviour

Two separate transactions are able to execute.

Steps To Reproduce

Run the test in https://github.com/lightbody/micronaut-data-4-tx-issue

Environment Information

No response

Example Application

https://github.com/lightbody/micronaut-data-4-tx-issue

Version

4.x

[lightbody]

@graemerocher @dstepanov Any suggested workarounds for this I can apply now? Our Micronaut 3->4 upgrade PR is 6 weeks old now and I'm worried we may have to throw it out and start over again if it ages too much longer. This issue is the only thing blocking our upgrade. Is there any way to reverse the effects of ExecutorServiceInstrumenter or perhaps force the transaction code to fetch a new connection / start a new transaction if it detects that it has a handle on a closed connection?

[graemerocher]

will look in the new year

[lightbody]

Just an update on our end: we worked around it by simply not using Micronaut-managed ExecutorService in our code. It was a massive effort, but we:

• Build our own ReclaimExecutor that wraps our own builtExecutorService that Micronaut doesn't "see"; this prevents all the propagation from kicking in and messing up transactions.
• Wrote our own ReclaimExecutorMetricsBinder that re-creates the logic in ExecutorServiceMetricsBinder so we retain the important observability we've grown to love and depend on.
• Removed all use of @Async.
• Removed most use of @Scheduled, but kept in place in a few places. In those places we made sure we immediately shifted the work over to our own ReclaimExecutor before any database activity took place.
• Replaced all use of ApplicationContext.publishEventAsync() to instead push work the appropriate ReclaimExecutor and then call ApplicationContext.publishEvent().

So at this point the issue isn't burning for us, but wanted to share the work we had to do to get around this in case anyone else is impacted. We will let you know if we find new issues that crop up related to this.