Flaky-test: DeferredSyncTest.testForceWillAdvanceLacOnlyUpToLastAcknoledgedWrite

[lhotari]

DeferredSyncTest.testForceWillAdvanceLacOnlyUpToLastAcknoledgedWrite is flaky. The build times out since the test never finishes. This happens quite often.

"main" #1 prio=5 os_prio=0 cpu=1865.56ms elapsed=1477.95s tid=0x00007f34b4028000 nid=0x149f5 waiting on condition  [0x00007f34bcffc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
	- parking to wait for  <0x000000008ad136e8> (a java.util.concurrent.CompletableFuture$Signaller)
	at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:234)
	at java.util.concurrent.CompletableFuture$Signaller.block([email protected]/CompletableFuture.java:1798)
	at java.util.concurrent.ForkJoinPool.managedBlock([email protected]/ForkJoinPool.java:3128)
	at java.util.concurrent.CompletableFuture.timedGet([email protected]/CompletableFuture.java:1871)
	at java.util.concurrent.CompletableFuture.get([email protected]/CompletableFuture.java:2027)
	at org.apache.bookkeeper.common.concurrent.FutureUtils.result(FutureUtils.java:80)
	at org.apache.bookkeeper.common.concurrent.FutureUtils.result(FutureUtils.java:61)
	at org.apache.bookkeeper.client.DeferredSyncTest.testForceWillAdvanceLacOnlyUpToLastAcknoledgedWrite(DeferredSyncTest.java:178)

Example failure: https://github.com/apache/bookkeeper/actions/runs/7776777094/job/21204429600?pr=4196#step:12:422

[lhotari]

@eolivelli Any ideas why this test DeferredSyncTest.testForceWillAdvanceLacOnlyUpToLastAcknoledgedWrite hangs forever?

[eolivelli]

Looking at the code I cannot find any obvious problem.
Maybe in resumeBookieWriteAcks things are not working properly ? we are missing some callback

we should reproduce the problem locally

[tmzk1005]

Maybe a thread synchronization problem :

code-block-1 and code-block-3 runs in main thread, but code-block-2 and code-block-4 runs in another thread, when the execution order is 1 -> 2 -> 3 -> 4. callback has no chance to run, so result(forceResult); blocked forever.

[tmzk1005]

cc @eolivelli, please review my code to see if I fix #4200 correctly. Thanks!