Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vertx zookeeper error #63

Open
mscofy opened this issue Nov 8, 2017 · 10 comments
Open

Vertx zookeeper error #63

mscofy opened this issue Nov 8, 2017 · 10 comments

Comments

@mscofy
Copy link

mscofy commented Nov 8, 2017

I am using vertx with zookeeper as my cluster manager. Suddenly I started getting this excpetion in my logs:

ERROR io.vertx.spi.cluster.zookeeper.ZookeeperClusterManager - Failed to handle memberRemoved
io.vertx.core.VertxException: java.io.InvalidClassException: io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap$KeyValue; local class incompatible: stream classdesc serialVersionUID = -472487054773491211, local class serialVersionUID = -6066652933022630380
at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.get(ZKSyncMap.java:95)
How can I fix it?

Regards, Ido

@stream-iori
Copy link
Contributor

Hi
Do you upgrade vert.x from old version to the latest?
which version of vert.x your running.

Thanks.

@mscofy
Copy link
Author

mscofy commented Nov 9, 2017

Yes,i upgrade vext.x from 3.4.2 to 3.5.0. Now i'm running version 3.5.0

@stream-iori
Copy link
Contributor

We using Java serialization to save object into the zk, this issue would be happen if there are multi version vertx-zookeeper in one cluster, you have to restart all the verticle with latest version of vertx-zookeeper.
I hope could refactor this method in the next release.

@aruis
Copy link

aruis commented Dec 28, 2018

I also encountered this problem. And I also used a different version of vertx-zookeeper.

[ Curator-PathChildrenCache-0 ] - [ ERROR ] [ i.v.spi.cluster.zookeeper.ZookeeperClusterManager : 167 ] - Failed to handle memberRemoved
io.vertx.core.json.DecodeException: Failed to decode: null
	at io.vertx.core.json.Json.decodeValue(Json.java:124)
	at io.vertx.core.json.JsonObject.fromJson(JsonObject.java:956)
	at io.vertx.core.json.JsonObject.<init>(JsonObject.java:48)
	at io.vertx.core.impl.HAManager.nodeLeft(HAManager.java:322)
	at io.vertx.core.impl.HAManager.access$100(HAManager.java:97)
	at io.vertx.core.impl.HAManager$1.nodeLeft(HAManager.java:150)
	at io.vertx.spi.cluster.zookeeper.ZookeeperClusterManager.childEvent(ZookeeperClusterManager.java:409)
	at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:522)

Now I try to upgrade to the latest version of vertx-zookeeper, and I will give feedback after the results.

@ijl20
Copy link

ijl20 commented May 15, 2020

In all the years we've been running vertx, first with Hazelcast and now with Zookeeper, this failure to connect to a running cluster unless you have the exact same minor version of vertx-hazelcast or vertx-zookeeper as every other verticle in the entire distributed system has been the major pain in the ass and IMHO an odd design choice. We heavily use eventbus message passing between verticles but don't attempt to share state - it might be more effective to use an external messaging service and not cluster the verticles. Of course this is totally doable but seems to defeat what I took as a core capability of Vertx.

@melaraj2
Copy link

melaraj2 commented Jun 20, 2020

I have the same version across the entire cluster, but this issue occurs when I redeploy one of the members. So far the only way to resolve it is to redeploy every node in the cluster. I am currently on version 3.9.1.

Is there anyone looking into this issue. I agree with ijl20. this is a severe problem, and like his system, we are only using clustering for event bus message passing.

This is my exception that i am seeing:

2020-06-20T10:24:16 - SEVERE: Failed to handle memberRemoved
2020-06-20T10:24:16 - io.vertx.core.VertxException: io.vertx.core.VertxException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /prod.backendApi/syncMap/__vertx.haInfo/aa7b8e23-5547-4135-981c-a7e2482cf149
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.keySet(ZKSyncMap.java:156)
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.entrySet(ZKSyncMap.java:180)
2020-06-20T10:24:16 - 	at io.vertx.core.impl.HAManager.nodeLeft(HAManager.java:320)
2020-06-20T10:24:16 - 	at io.vertx.core.impl.HAManager.access$100(HAManager.java:97)
2020-06-20T10:24:16 - 	at io.vertx.core.impl.HAManager$1.nodeLeft(HAManager.java:150)
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.ZookeeperClusterManager.childEvent(ZookeeperClusterManager.java:409)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:522)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93)
2020-06-20T10:24:16 - 	at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:514)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-06-20T10:24:16 - 	at java.base/java.lang.Thread.run(Thread.java:834)
2020-06-20T10:24:16 - Caused by: io.vertx.core.VertxException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /prod.backendApi/syncMap/__vertx.haInfo/aa7b8e23-5547-4135-981c-a7e2482cf149
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.lambda$keySet$2(ZKSyncMap.java:152)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
2020-06-20T10:24:16 - 	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.keySet(ZKSyncMap.java:154)
2020-06-20T10:24:16 - 	... 20 more
2020-06-20T10:24:16 - Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /prod.backendApi/syncMap/__vertx.haInfo/aa7b8e23-5547-4135-981c-a7e2482cf149
2020-06-20T10:24:16 - 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:114)
2020-06-20T10:24:16 - 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
2020-06-20T10:24:16 - 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:310)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:299)
2020-06-20T10:24:16 - 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:296)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:287)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:34)
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.lambda$keySet$2(ZKSyncMap.java:149)
2020-06-20T10:24:16 - 	... 28 more
2020-06-20T10:24:22 - Jun 20, 2020 2:24:22 PM io.vertx.core.eventbus.impl.clustered.ClusteredEventBus
2020-06-20T10:24:22 - WARNING: Error removing subs
2020-06-20T10:24:22 - java.io.EOFException
2020-06-20T10:24:22 - 	at java.base/java.io.DataInputStream.readBoolean(DataInputStream.java:249)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKMap.asObject(ZKMap.java:110)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.lambda$null$22(ZKAsyncMultiMap.java:195)
2020-06-20T10:24:22 - 	at java.base/java.util.Optional.ifPresent(Optional.java:183)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.lambda$null$23(ZKAsyncMultiMap.java:193)
2020-06-20T10:24:22 - 	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.lambda$null$24(ZKAsyncMultiMap.java:189)
2020-06-20T10:24:22 - 	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.lambda$removeAllMatching$26(ZKAsyncMultiMap.java:187)
2020-06-20T10:24:22 - 	at java.base/java.util.Optional.ifPresent(Optional.java:183)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.removeAllMatching(ZKAsyncMultiMap.java:186)
2020-06-20T10:24:22 - 	at io.vertx.core.eventbus.impl.clustered.ClusteredEventBus.lambda$setClusterViewChangedHandler$12(ClusteredEventBus.java:273)
2020-06-20T10:24:22 - 	at io.vertx.core.impl.HAManager.lambda$checkSubs$12(HAManager.java:520)
2020-06-20T10:24:22 - 	at io.vertx.core.impl.HAManager.lambda$runOnContextAndWait$13(HAManager.java:529)
2020-06-20T10:24:22 - 	at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
2020-06-20T10:24:22 - 	at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
2020-06-20T10:24:22 - 	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
2020-06-20T10:24:22 - 	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
2020-06-20T10:24:22 - 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
2020-06-20T10:24:22 - 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
2020-06-20T10:24:22 - 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2020-06-20T10:24:22 - 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2020-06-20T10:24:22 - 	at java.base/java.lang.Thread.run(Thread.java:834)


@stream-iori
Copy link
Contributor

I have the same version across the entire cluster, but this issue occurs when I redeploy one of the members. So far the only way to resolve it is to redeploy every node in the cluster. I am currently on version 3.9.1.

Is there anyone looking into this issue. I agree with ijl20. this is a severe problem, and like his system, we are only using clustering for event bus message passing.

This is my exception that i am seeing:

2020-06-20T10:24:16 - SEVERE: Failed to handle memberRemoved
2020-06-20T10:24:16 - io.vertx.core.VertxException: io.vertx.core.VertxException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /prod.backendApi/syncMap/__vertx.haInfo/aa7b8e23-5547-4135-981c-a7e2482cf149
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.keySet(ZKSyncMap.java:156)
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.entrySet(ZKSyncMap.java:180)
2020-06-20T10:24:16 - 	at io.vertx.core.impl.HAManager.nodeLeft(HAManager.java:320)
2020-06-20T10:24:16 - 	at io.vertx.core.impl.HAManager.access$100(HAManager.java:97)
2020-06-20T10:24:16 - 	at io.vertx.core.impl.HAManager$1.nodeLeft(HAManager.java:150)
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.ZookeeperClusterManager.childEvent(ZookeeperClusterManager.java:409)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:522)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93)
2020-06-20T10:24:16 - 	at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:514)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2020-06-20T10:24:16 - 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2020-06-20T10:24:16 - 	at java.base/java.lang.Thread.run(Thread.java:834)
2020-06-20T10:24:16 - Caused by: io.vertx.core.VertxException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /prod.backendApi/syncMap/__vertx.haInfo/aa7b8e23-5547-4135-981c-a7e2482cf149
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.lambda$keySet$2(ZKSyncMap.java:152)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
2020-06-20T10:24:16 - 	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2020-06-20T10:24:16 - 	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.keySet(ZKSyncMap.java:154)
2020-06-20T10:24:16 - 	... 20 more
2020-06-20T10:24:16 - Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /prod.backendApi/syncMap/__vertx.haInfo/aa7b8e23-5547-4135-981c-a7e2482cf149
2020-06-20T10:24:16 - 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:114)
2020-06-20T10:24:16 - 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
2020-06-20T10:24:16 - 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:310)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:299)
2020-06-20T10:24:16 - 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:296)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:287)
2020-06-20T10:24:16 - 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:34)
2020-06-20T10:24:16 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKSyncMap.lambda$keySet$2(ZKSyncMap.java:149)
2020-06-20T10:24:16 - 	... 28 more
2020-06-20T10:24:22 - Jun 20, 2020 2:24:22 PM io.vertx.core.eventbus.impl.clustered.ClusteredEventBus
2020-06-20T10:24:22 - WARNING: Error removing subs
2020-06-20T10:24:22 - java.io.EOFException
2020-06-20T10:24:22 - 	at java.base/java.io.DataInputStream.readBoolean(DataInputStream.java:249)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKMap.asObject(ZKMap.java:110)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.lambda$null$22(ZKAsyncMultiMap.java:195)
2020-06-20T10:24:22 - 	at java.base/java.util.Optional.ifPresent(Optional.java:183)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.lambda$null$23(ZKAsyncMultiMap.java:193)
2020-06-20T10:24:22 - 	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.lambda$null$24(ZKAsyncMultiMap.java:189)
2020-06-20T10:24:22 - 	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.lambda$removeAllMatching$26(ZKAsyncMultiMap.java:187)
2020-06-20T10:24:22 - 	at java.base/java.util.Optional.ifPresent(Optional.java:183)
2020-06-20T10:24:22 - 	at io.vertx.spi.cluster.zookeeper.impl.ZKAsyncMultiMap.removeAllMatching(ZKAsyncMultiMap.java:186)
2020-06-20T10:24:22 - 	at io.vertx.core.eventbus.impl.clustered.ClusteredEventBus.lambda$setClusterViewChangedHandler$12(ClusteredEventBus.java:273)
2020-06-20T10:24:22 - 	at io.vertx.core.impl.HAManager.lambda$checkSubs$12(HAManager.java:520)
2020-06-20T10:24:22 - 	at io.vertx.core.impl.HAManager.lambda$runOnContextAndWait$13(HAManager.java:529)
2020-06-20T10:24:22 - 	at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
2020-06-20T10:24:22 - 	at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
2020-06-20T10:24:22 - 	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
2020-06-20T10:24:22 - 	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
2020-06-20T10:24:22 - 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
2020-06-20T10:24:22 - 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
2020-06-20T10:24:22 - 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2020-06-20T10:24:22 - 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2020-06-20T10:24:22 - 	at java.base/java.lang.Thread.run(Thread.java:834)

This exception should have been swallow and make a WARNING to log.
I am workng on vertx4-zookeeper, and this exception should convert to warning log

@melaraj2
Copy link

melaraj2 commented Jun 21, 2020

The problem is not just the exception, in my case I have a JVM instance that calls eventBus.request to an address being consumed by a set of JVMs, when I restart the consumers, the requests to that address fail with timeout exceptions. It appears that the only way to resolve it is by also restarting the requester JVMs on other instances. In my case, it is a normal course of business to update the consumers in the cluster. I imagine this is a common thing.

All of this runs on AWS Fargate, so its important to note that IP addresses change when i restart the service.

@melaraj2
Copy link

the sending Vertx instances continue to attempt to send the request to the old IP address that went away. This is permanent, there is no recovery unless I restart the sender.

@vietj
Copy link
Contributor

vietj commented Jun 21, 2020 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

6 participants