exception on shutdown of a clustered vertx instance

[radai-rosenblatt]

im constructing a simple zookeper-backed clustered vertx, like so:

VertxOptions options = new VertxOptions();
ClusterManager clusterManager = new ZookeeperClusterManager(zkConfig);
options.setClusterManager(clusterManager);
EventBusOptions eventBusOptions = options.getEventBusOptions();
eventBusOptions.setClustered(true);
Vertx.clusteredVertx(options, ...);

seems to work fine, but when i shut it down, i get this:

10:01:15.852 [Curator-PathChildrenCache-0] ERROR org.apache.curator.framework.recipes.cache.PathChildrenCache - 
java.lang.IllegalStateException: instance must be started before calling this method
	at org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:176) ~[curator-client-2.12.0.jar:?]
	at org.apache.curator.framework.imps.CuratorFrameworkImpl.getChildren(CuratorFrameworkImpl.java:391) ~[curator-framework-2.12.0.jar:?]
	at org.apache.curator.framework.recipes.cache.PathChildrenCache.refresh(PathChildrenCache.java:508) [curator-recipes-2.12.0.jar:?]
	at org.apache.curator.framework.recipes.cache.RefreshOperation.invoke(RefreshOperation.java:35) ~[curator-recipes-2.12.0.jar:?]
	at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773) [curator-recipes-2.12.0.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172]
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) [?:1.8.0_172]
	at java.util.concurrent.FutureTask.run(FutureTask.java) [?:1.8.0_172]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172]
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) [?:1.8.0_172]
	at java.util.concurrent.FutureTask.run(FutureTask.java) [?:1.8.0_172]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
10:01:15.859 [ProcessThread(sid:0 cport:40734):] INFO  org.apache.zookeeper.server.PrepRequestProcessor - Processed session termination for sessionid: 0x1000a8b10eb0000
Jul 17, 2019 10:01:15 AM io.vertx.core.eventbus.impl.clustered.ClusteredEventBus
SEVERE: Failed to remove sub
java.lang.IllegalStateException: instance must be started before calling this method
	at org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:176)
	at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkExists(CuratorFrameworkImpl.java:367)
	at io.vertx.spi.cluster.zookeeper.impl.ZKMap.lambda$checkExists$5(ZKMap.java:198)
	at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:749)
	at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:522)
	at org.apache.curator.framework.imps.SyncBuilderImpl$1.processResult(SyncBuilderImpl.java:108)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:637)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508)

the offending line is in curator checkExists():

Preconditions.checkState(getState() == CuratorFrameworkState.STARTED, "instance must be started before calling this method");

state is closed at the time of this check.

this happens for both 3.7.1 and 3.8.0

[stream-iori]

Hi radai
That is a flaw on vertx-zookeeper, i did not check Curator Client State while Curator have closed with zookeeper, and i have no way to tell the Vertx that Client is break with Server.
I will try to fix such flaw in Vertx4 by checking state of Curator.

[radai-rosenblatt]

@stream1984 - are there builds of vertx-zk 4.* that i can try somewhere? also (in my opinion) curator's dependency on unreleased "beta" versions of ZK makes it unfit for production use - have you considered using the "raw" zk client ?

[stream-iori]

@radai-rosenblatt
you can try to update version <curator.version>2.12.0</curator.version> in your branch, and i will also upgrade curator to the 4.x in next month with Vert.x4.
You could exclude version of ZK-beta and include ZK-3.5.5 in your pom.
well, using raw zk-client would make more code for some trivia code, i have no enough time to do that, maybe you could provide some help :)

[radai-rosenblatt]

latest curator "scares" me because my workplace doesnt run zk 3.5 in production - i dont know of any org that does. they claim to be runtime compatible with 3.4, but ....