Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SessionExpiredException: KeeperErrorCode = Session expired for /stm/atoms/jepsen #533

Open
jack-tang1 opened this issue Mar 13, 2022 · 2 comments
Open

SessionExpiredException: KeeperErrorCode = Session expired for /stm/atoms/jepsen #533

jack-tang1 opened this issue Mar 13, 2022 · 2 comments

Comments

@jack-tang1
Copy link

Hello,i modify dockerfile for debian 10,and use the Zookeeper Debian package for installation.The zookeeper version is 3.4.13-2--1.
Execute zookeeper case in current repo,get the stack trace.

WARN [2022-03-13 03:19:08,301] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041)
WARN [2022-03-13 03:19:08,306] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041)
INFO [2022-03-13 03:19:08,410] jepsen node n1-SendThread(jepsen-n1.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session establishment complete on server jepsen-n1.jepsen_jepsen/172.19.0.4:2181, sessionid = 0x143a37d150004, negotiated timeout = 5000
INFO [2022-03-13 03:19:08,412] jepsen node n2-SendThread(jepsen-n2.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session establishment complete on server jepsen-n2.jepsen_jepsen/172.19.0.6:2181, sessionid = 0x10143a37d220004, negotiated timeout = 5000
INFO [2022-03-13 03:19:08,428] jepsen node n3-SendThread(jepsen-n3.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session establishment complete on server jepsen-n3.jepsen_jepsen/172.19.0.3:2181, sessionid = 0x20143a37d400005, negotiated timeout = 5000
INFO [2022-03-13 03:19:09,427] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n5.jepsen_jepsen/172.19.0.2:2181
INFO [2022-03-13 03:19:09,428] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n4.jepsen_jepsen/172.19.0.5:2181
WARN [2022-03-13 03:19:09,430] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041)
WARN [2022-03-13 03:19:09,434] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041)
INFO [2022-03-13 03:19:10,531] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n4.jepsen_jepsen/172.19.0.5:2181
WARN [2022-03-13 03:19:10,533] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041)
INFO [2022-03-13 03:19:10,534] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n5.jepsen_jepsen/172.19.0.2:2181
WARN [2022-03-13 03:19:10,537] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041)
INFO [2022-03-13 03:19:11,634] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n4.jepsen_jepsen/172.19.0.5:2181
WARN [2022-03-13 03:19:11,635] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041)
INFO [2022-03-13 03:19:11,638] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n5.jepsen_jepsen/172.19.0.2:2181
WARN [2022-03-13 03:19:11,639] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041)
INFO [2022-03-13 03:19:12,736] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n4.jepsen_jepsen/172.19.0.5:2181
WARN [2022-03-13 03:19:12,738] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041)
INFO [2022-03-13 03:19:12,740] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n5.jepsen_jepsen/172.19.0.2:2181
WARN [2022-03-13 03:19:12,743] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1041)
INFO [2022-03-13 03:19:13,840] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n4.jepsen_jepsen/172.19.0.5:2181
INFO [2022-03-13 03:19:13,841] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n4.jepsen_jepsen/172.19.0.5:2181, initiating session
INFO [2022-03-13 03:19:13,849] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n5.jepsen_jepsen/172.19.0.2:2181
INFO [2022-03-13 03:19:13,852] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n5.jepsen_jepsen/172.19.0.2:2181, initiating session
INFO [2022-03-13 03:19:13,989] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
INFO [2022-03-13 03:19:14,024] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
INFO [2022-03-13 03:19:15,250] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n5.jepsen_jepsen/172.19.0.2:2181
INFO [2022-03-13 03:19:15,251] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n5.jepsen_jepsen/172.19.0.2:2181, initiating session
INFO [2022-03-13 03:19:15,300] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session establishment complete on server jepsen-n5.jepsen_jepsen/172.19.0.2:2181, sessionid = 0x40143b19ff50000, negotiated timeout = 5000
INFO [2022-03-13 03:19:15,543] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n4.jepsen_jepsen/172.19.0.5:2181
INFO [2022-03-13 03:19:15,544] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n4.jepsen_jepsen/172.19.0.5:2181, initiating session
INFO [2022-03-13 03:19:15,565] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session establishment complete on server jepsen-n4.jepsen_jepsen/172.19.0.5:2181, sessionid = 0x30143b1a0c10000, negotiated timeout = 5000
INFO [2022-03-13 03:19:15,753] jepsen worker 4 - jepsen.util 4	:invoke	:read	nil
INFO [2022-03-13 03:19:15,756] jepsen worker 4 - org.apache.zookeeper.ZooKeeper Initiating client connection, connectString=n5 sessionTimeout=5000 watcher=zookeeper.internal$make_watcher$reify__440@662e84f0
INFO [2022-03-13 03:19:15,762] jepsen worker 4-SendThread() - org.apache.zookeeper.ClientCnxn Opening socket connection to server /172.19.0.2:2181
INFO [2022-03-13 03:19:15,766] jepsen worker 4-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n5.jepsen_jepsen/172.19.0.2:2181, initiating session
INFO [2022-03-13 03:19:15,779] jepsen worker 4-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session establishment complete on server jepsen-n5.jepsen_jepsen/172.19.0.2:2181, sessionid = 0x40143b19ff50001, negotiated timeout = 5000
INFO [2022-03-13 03:19:15,958] jepsen worker 4 - jepsen.util 4	:ok	:read	0
INFO [2022-03-13 03:19:16,021] jepsen worker 1 - jepsen.util 1	:invoke	:cas	[2 0]
INFO [2022-03-13 03:19:16,022] jepsen worker 1 - org.apache.zookeeper.ZooKeeper Initiating client connection, connectString=n2 sessionTimeout=5000 watcher=zookeeper.internal$make_watcher$reify__440@7453aea6
INFO [2022-03-13 03:19:16,025] jepsen worker 1-SendThread() - org.apache.zookeeper.ClientCnxn Opening socket connection to server /172.19.0.6:2181
INFO [2022-03-13 03:19:16,027] jepsen worker 1-SendThread(jepsen-n2.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n2.jepsen_jepsen/172.19.0.6:2181, initiating session
INFO [2022-03-13 03:19:16,039] jepsen worker 1-SendThread(jepsen-n2.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session establishment complete on server jepsen-n2.jepsen_jepsen/172.19.0.6:2181, sessionid = 0x10143a37d220005, negotiated timeout = 5000
INFO [2022-03-13 03:19:16,217] jepsen worker 1 - jepsen.util 1	:fail	:cas	[2 0]
INFO [2022-03-13 03:19:17,751] jepsen worker 0 - jepsen.util 0	:invoke	:read	nil
INFO [2022-03-13 03:19:17,752] jepsen worker 0 - org.apache.zookeeper.ZooKeeper Initiating client connection, connectString=n1 sessionTimeout=5000 watcher=zookeeper.internal$make_watcher$reify__440@6a217d3c
INFO [2022-03-13 03:19:17,755] jepsen worker 0-SendThread() - org.apache.zookeeper.ClientCnxn Opening socket connection to server /172.19.0.4:2181
INFO [2022-03-13 03:19:17,760] jepsen worker 0-SendThread(jepsen-n1.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n1.jepsen_jepsen/172.19.0.4:2181, initiating session
INFO [2022-03-13 03:19:17,769] jepsen worker 0-SendThread(jepsen-n1.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session establishment complete on server jepsen-n1.jepsen_jepsen/172.19.0.4:2181, sessionid = 0x143a37d150005, negotiated timeout = 5000
INFO [2022-03-13 03:19:17,866] jepsen worker 0 - jepsen.util 0	:ok	:read	0
INFO [2022-03-13 03:19:17,990] jepsen worker 1 - jepsen.util 1	:invoke	:cas	[0 2]
INFO [2022-03-13 03:19:18,049] jepsen worker 1 - jepsen.util 1	:ok	:cas	[0 2]
INFO [2022-03-13 03:19:18,487] jepsen worker 2 - jepsen.util 2	:invoke	:write	4
INFO [2022-03-13 03:19:18,488] jepsen worker 2 - org.apache.zookeeper.ZooKeeper Initiating client connection, connectString=n3 sessionTimeout=5000 watcher=zookeeper.internal$make_watcher$reify__440@6dd570c8
INFO [2022-03-13 03:19:18,490] jepsen worker 2-SendThread() - org.apache.zookeeper.ClientCnxn Opening socket connection to server /172.19.0.3:2181
INFO [2022-03-13 03:19:18,494] jepsen worker 2-SendThread(jepsen-n3.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n3.jepsen_jepsen/172.19.0.3:2181, initiating session
INFO [2022-03-13 03:19:18,504] jepsen worker 2-SendThread(jepsen-n3.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session establishment complete on server jepsen-n3.jepsen_jepsen/172.19.0.3:2181, sessionid = 0x20143a37d400006, negotiated timeout = 5000
INFO [2022-03-13 03:19:18,646] jepsen worker 2 - jepsen.util 2	:ok	:write	4
INFO [2022-03-13 03:19:19,212] jepsen worker 2 - jepsen.util 2	:invoke	:read	nil
INFO [2022-03-13 03:19:19,217] jepsen worker 2 - jepsen.util 2	:ok	:read	4
INFO [2022-03-13 03:19:20,754] jepsen worker nemesis - jepsen.util :nemesis	:info	:start	nil
INFO [2022-03-13 03:19:21,144] jepsen worker 0 - jepsen.util 0	:invoke	:cas	[4 2]
INFO [2022-03-13 03:19:21,191] jepsen worker nemesis - jepsen.util :nemesis	:info	:start	[:isolated {"n1" #{"n2" "n5" "n4"}, "n3" #{"n2" "n5" "n4"}, "n2" #{"n1" "n3"}, "n5" #{"n1" "n3"}, "n4" #{"n1" "n3"}}]
INFO [2022-03-13 03:19:22,737] jepsen worker 4 - jepsen.util 4	:invoke	:cas	[3 2]
INFO [2022-03-13 03:19:23,629] jepsen node n3-SendThread(jepsen-n3.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Client session timed out, have not heard from server in 3334ms for sessionid 0x20143a37d400005, closing socket connection and attempting reconnect
INFO [2022-03-13 03:19:23,635] jepsen node n1-SendThread(jepsen-n1.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Client session timed out, have not heard from server in 3335ms for sessionid 0x143a37d150004, closing socket connection and attempting reconnect
INFO [2022-03-13 03:19:23,687] jepsen worker 2 - jepsen.util 2	:invoke	:cas	[4 2]
INFO [2022-03-13 03:19:23,885] jepsen worker 1 - jepsen.util 1	:invoke	:cas	[3 2]
INFO [2022-03-13 03:19:24,219] jepsen worker 2-SendThread(jepsen-n3.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Client session timed out, have not heard from server in 3334ms for sessionid 0x20143a37d400006, closing socket connection and attempting reconnect
WARN [2022-03-13 03:19:24,323] jepsen worker 2 - jepsen.generator.interpreter Process 2 crashed
java.lang.IllegalMonitorStateException: null
	at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:372)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1302)
	at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1147)
	at java.base/jdk.internal.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:438)
	at avout.locks.ZKDistributedReentrantLock.unlock(locks.clj:130)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:438)
	at avout.atoms.DistributedAtom.swap(atoms.clj:58)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102)
	at avout.core$swap_BANG__BANG_.invokeStatic(core.clj:107)
	at avout.core$swap_BANG__BANG_.doInvoke(core.clj:105)
	at clojure.lang.RestFn.invoke(RestFn.java:425)
	at jepsen.zookeeper$client$reify__257$fn__258.invoke(zookeeper.clj:98)
	at clojure.core$binding_conveyor_fn$fn__5739.invoke(core.clj:2030)
	at clojure.lang.AFn.call(AFn.java:18)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
INFO [2022-03-13 03:19:24,492] jepsen worker 0-SendThread(jepsen-n1.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Client session timed out, have not heard from server in 3335ms for sessionid 0x143a37d150005, closing socket connection and attempting reconnect
WARN [2022-03-13 03:19:24,595] jepsen worker 0 - jepsen.generator.interpreter Process 0 crashed
java.lang.IllegalMonitorStateException: null
	at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:372)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1302)
	at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1147)
	at java.base/jdk.internal.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:438)
	at avout.locks.ZKDistributedReentrantLock.unlock(locks.clj:130)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:438)
	at avout.atoms.DistributedAtom.swap(atoms.clj:58)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102)
	at avout.core$swap_BANG__BANG_.invokeStatic(core.clj:107)
	at avout.core$swap_BANG__BANG_.doInvoke(core.clj:105)
	at clojure.lang.RestFn.invoke(RestFn.java:425)
	at jepsen.zookeeper$client$reify__257$fn__258.invoke(zookeeper.clj:98)
	at clojure.core$binding_conveyor_fn$fn__5739.invoke(core.clj:2030)
	at clojure.lang.AFn.call(AFn.java:18)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
INFO [2022-03-13 03:19:25,296] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Client session timed out, have not heard from server in 3333ms for sessionid 0x40143b19ff50000, closing socket connection and attempting reconnect
INFO [2022-03-13 03:19:25,455] jepsen node n1-SendThread(jepsen-n1.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n1.jepsen_jepsen/172.19.0.4:2181
INFO [2022-03-13 03:19:25,456] jepsen node n1-SendThread(jepsen-n1.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n1.jepsen_jepsen/172.19.0.4:2181, initiating session
INFO [2022-03-13 03:19:25,472] jepsen node n1-SendThread(jepsen-n1.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session establishment complete on server jepsen-n1.jepsen_jepsen/172.19.0.4:2181, sessionid = 0x143a37d150004, negotiated timeout = 5000
INFO [2022-03-13 03:19:25,594] jepsen worker 2-SendThread(jepsen-n3.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n3.jepsen_jepsen/172.19.0.3:2181
INFO [2022-03-13 03:19:25,595] jepsen worker 2-SendThread(jepsen-n3.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n3.jepsen_jepsen/172.19.0.3:2181, initiating session
INFO [2022-03-13 03:19:25,599] jepsen worker 2-SendThread(jepsen-n3.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session establishment complete on server jepsen-n3.jepsen_jepsen/172.19.0.3:2181, sessionid = 0x20143a37d400006, negotiated timeout = 5000
INFO [2022-03-13 03:19:25,723] jepsen node n3-SendThread(jepsen-n3.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n3.jepsen_jepsen/172.19.0.3:2181
INFO [2022-03-13 03:19:25,725] jepsen node n3-SendThread(jepsen-n3.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n3.jepsen_jepsen/172.19.0.3:2181, initiating session
INFO [2022-03-13 03:19:25,736] jepsen node n3-SendThread(jepsen-n3.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session establishment complete on server jepsen-n3.jepsen_jepsen/172.19.0.3:2181, sessionid = 0x20143a37d400005, negotiated timeout = 5000
INFO [2022-03-13 03:19:25,855] jepsen worker 0 - jepsen.util 5	:invoke	:read	nil
INFO [2022-03-13 03:19:26,084] jepsen worker 4-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Client session timed out, have not heard from server in 3335ms for sessionid 0x40143b19ff50001, closing socket connection and attempting reconnect
INFO [2022-03-13 03:19:26,132] jepsen worker 0-SendThread(jepsen-n1.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n1.jepsen_jepsen/172.19.0.4:2181
INFO [2022-03-13 03:19:26,133] jepsen worker 0-SendThread(jepsen-n1.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n1.jepsen_jepsen/172.19.0.4:2181, initiating session
INFO [2022-03-13 03:19:26,144] jepsen worker 0-SendThread(jepsen-n1.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session establishment complete on server jepsen-n1.jepsen_jepsen/172.19.0.4:2181, sessionid = 0x143a37d150005, negotiated timeout = 5000
WARN [2022-03-13 03:19:26,186] jepsen worker 4 - jepsen.generator.interpreter Process 4 crashed
java.lang.IllegalMonitorStateException: null
	at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:372)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1302)
	at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1147)
	at java.base/jdk.internal.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:438)
	at avout.locks.ZKDistributedReentrantLock.unlock(locks.clj:130)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:438)
	at avout.atoms.DistributedAtom.swap(atoms.clj:58)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102)
	at avout.core$swap_BANG__BANG_.invokeStatic(core.clj:107)
	at avout.core$swap_BANG__BANG_.doInvoke(core.clj:105)
	at clojure.lang.RestFn.invoke(RestFn.java:425)
	at jepsen.zookeeper$client$reify__257$fn__258.invoke(zookeeper.clj:98)
	at clojure.core$binding_conveyor_fn$fn__5739.invoke(core.clj:2030)
	at clojure.lang.AFn.call(AFn.java:18)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
INFO [2022-03-13 03:19:26,195] jepsen worker nemesis - jepsen.util :nemesis	:info	:stop	nil
INFO [2022-03-13 03:19:26,440] jepsen worker nemesis - jepsen.util :nemesis	:info	:stop	:network-healed
INFO [2022-03-13 03:19:26,969] jepsen node n2-SendThread(jepsen-n2.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Client session timed out, have not heard from server in 3334ms for sessionid 0x10143a37d220004, closing socket connection and attempting reconnect
INFO [2022-03-13 03:19:27,232] jepsen worker 1-SendThread(jepsen-n2.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Client session timed out, have not heard from server in 3335ms for sessionid 0x10143a37d220005, closing socket connection and attempting reconnect
INFO [2022-03-13 03:19:27,312] jepsen worker 4 - jepsen.util 9	:invoke	:write	3
WARN [2022-03-13 03:19:27,335] jepsen worker 1 - jepsen.generator.interpreter Process 1 crashed
java.lang.IllegalMonitorStateException: null
	at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:372)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1302)
	at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1147)
	at java.base/jdk.internal.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:438)
	at avout.locks.ZKDistributedReentrantLock.unlock(locks.clj:130)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:438)
	at avout.atoms.DistributedAtom.swap(atoms.clj:58)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
	at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102)
	at avout.core$swap_BANG__BANG_.invokeStatic(core.clj:107)
	at avout.core$swap_BANG__BANG_.doInvoke(core.clj:105)
	at clojure.lang.RestFn.invoke(RestFn.java:425)
	at jepsen.zookeeper$client$reify__257$fn__258.invoke(zookeeper.clj:98)
	at clojure.core$binding_conveyor_fn$fn__5739.invoke(core.clj:2030)
	at clojure.lang.AFn.call(AFn.java:18)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
INFO [2022-03-13 03:19:27,390] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n5.jepsen_jepsen/172.19.0.2:2181
INFO [2022-03-13 03:19:27,392] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n5.jepsen_jepsen/172.19.0.2:2181, initiating session
INFO [2022-03-13 03:19:27,680] jepsen worker 4-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n5.jepsen_jepsen/172.19.0.2:2181
INFO [2022-03-13 03:19:27,681] jepsen worker 4-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n5.jepsen_jepsen/172.19.0.2:2181, initiating session
INFO [2022-03-13 03:19:27,966] jepsen worker 3 - jepsen.util 3	:invoke	:read	nil
INFO [2022-03-13 03:19:27,966] jepsen worker 3 - org.apache.zookeeper.ZooKeeper Initiating client connection, connectString=n4 sessionTimeout=5000 watcher=zookeeper.internal$make_watcher$reify__440@26a455ef
INFO [2022-03-13 03:19:27,980] jepsen worker 3-SendThread() - org.apache.zookeeper.ClientCnxn Opening socket connection to server /172.19.0.5:2181
INFO [2022-03-13 03:19:27,990] jepsen worker 3-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n4.jepsen_jepsen/172.19.0.5:2181, initiating session
ERROR [2022-03-13 03:19:28,057] jepsen worker 0-EventThread - org.apache.zookeeper.ClientCnxn Error while calling watcher
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /stm/atoms/jepsen
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
	at zookeeper$exists.invokeStatic(zookeeper.clj:96)
	at zookeeper$exists.doInvoke(zookeeper.clj:54)
	at clojure.lang.RestFn.invoke(RestFn.java:464)
	at avout.atoms.DistributedAtom.invalidateCache(atoms.clj:107)
	at avout.atoms.DistributedAtom$fn__1307.invoke(atoms.clj:108)
	at zookeeper.internal$make_watcher$reify__440.process(internal.clj:56)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
INFO [2022-03-13 03:19:28,070] jepsen worker 0 - org.apache.zookeeper.ZooKeeper Session: 0x143a37d150005 closed
INFO [2022-03-13 03:19:28,070] jepsen worker 0 - org.apache.zookeeper.ZooKeeper Initiating client connection, connectString=n1 sessionTimeout=5000 watcher=zookeeper.internal$make_watcher$reify__440@64e68e08
INFO [2022-03-13 03:19:28,071] jepsen worker 0-EventThread - org.apache.zookeeper.ClientCnxn EventThread shut down
INFO [2022-03-13 03:19:28,079] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Unable to read additional data from server sessionid 0x30143b1a0c10000, likely server has closed socket, closing socket connection and attempting reconnect
INFO [2022-03-13 03:19:28,080] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Unable to read additional data from server sessionid 0x40143b19ff50000, likely server has closed socket, closing socket connection and attempting reconnect
INFO [2022-03-13 03:19:28,081] jepsen worker 0-SendThread() - org.apache.zookeeper.ClientCnxn Opening socket connection to server /172.19.0.4:2181
INFO [2022-03-13 03:19:28,086] jepsen worker 3-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session establishment complete on server jepsen-n4.jepsen_jepsen/172.19.0.5:2181, sessionid = 0x30143b1a0c10001, negotiated timeout = 5000
INFO [2022-03-13 03:19:28,089] jepsen worker 0-SendThread(jepsen-n1.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n1.jepsen_jepsen/172.19.0.4:2181, initiating session
INFO [2022-03-13 03:19:28,096] jepsen worker 0-SendThread(jepsen-n1.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session establishment complete on server jepsen-n1.jepsen_jepsen/172.19.0.4:2181, sessionid = 0x143a37d150006, negotiated timeout = 5000
INFO [2022-03-13 03:19:28,130] jepsen node n2-SendThread(jepsen-n2.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n2.jepsen_jepsen/172.19.0.6:2181
INFO [2022-03-13 03:19:28,135] jepsen node n2-SendThread(jepsen-n2.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n2.jepsen_jepsen/172.19.0.6:2181, initiating session
INFO [2022-03-13 03:19:28,152] jepsen node n2-SendThread(jepsen-n2.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x10143a37d220004 has expired, closing socket connection
ERROR [2022-03-13 03:19:28,183] jepsen node n5-EventThread - org.apache.zookeeper.ClientCnxn Error while calling watcher
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /stm/atoms/jepsen
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
	at zookeeper$exists.invokeStatic(zookeeper.clj:96)
	at zookeeper$exists.doInvoke(zookeeper.clj:54)
	at clojure.lang.RestFn.invoke(RestFn.java:464)
	at avout.atoms.DistributedAtom.invalidateCache(atoms.clj:107)
	at avout.atoms.DistributedAtom$fn__1307.invoke(atoms.clj:108)
	at zookeeper.internal$make_watcher$reify__440.process(internal.clj:56)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
INFO [2022-03-13 03:19:28,203] jepsen worker 4 - org.apache.zookeeper.ZooKeeper Session: 0x40143b19ff50001 closed
INFO [2022-03-13 03:19:28,204] jepsen worker 4 - org.apache.zookeeper.ZooKeeper Initiating client connection, connectString=n5 sessionTimeout=5000 watcher=zookeeper.internal$make_watcher$reify__440@34e1e9af
ERROR [2022-03-13 03:19:28,204] jepsen worker 4-EventThread - org.apache.zookeeper.ClientCnxn Error while calling watcher
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /stm/atoms/jepsen
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
	at zookeeper$exists.invokeStatic(zookeeper.clj:96)
	at zookeeper$exists.doInvoke(zookeeper.clj:54)
	at clojure.lang.RestFn.invoke(RestFn.java:464)
	at avout.atoms.DistributedAtom.invalidateCache(atoms.clj:107)
	at avout.atoms.DistributedAtom$fn__1307.invoke(atoms.clj:108)
	at zookeeper.internal$make_watcher$reify__440.process(internal.clj:56)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
INFO [2022-03-13 03:19:28,205] jepsen worker 4-EventThread - org.apache.zookeeper.ClientCnxn EventThread shut down
INFO [2022-03-13 03:19:28,206] jepsen worker 4-SendThread() - org.apache.zookeeper.ClientCnxn Opening socket connection to server /172.19.0.2:2181
ERROR [2022-03-13 03:19:28,255] jepsen node n2-EventThread - org.apache.zookeeper.ClientCnxn Error while calling watcher
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /stm/atoms/jepsen
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
	at zookeeper$exists.invokeStatic(zookeeper.clj:96)
	at zookeeper$exists.doInvoke(zookeeper.clj:54)
	at clojure.lang.RestFn.invoke(RestFn.java:464)
	at avout.atoms.DistributedAtom.invalidateCache(atoms.clj:107)
	at avout.atoms.DistributedAtom$fn__1307.invoke(atoms.clj:108)
	at zookeeper.internal$make_watcher$reify__440.process(internal.clj:56)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
ERROR [2022-03-13 03:19:28,259] jepsen node n2-EventThread - org.apache.zookeeper.ClientCnxn Error while calling watcher
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /stm/atoms/jepsen
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
	at zookeeper$exists.invokeStatic(zookeeper.clj:96)
	at zookeeper$exists.doInvoke(zookeeper.clj:54)
	at clojure.lang.RestFn.invoke(RestFn.java:464)
	at avout.atoms.DistributedAtom.invalidateCache(atoms.clj:107)
	at avout.atoms.DistributedAtom$fn__1307.invoke(atoms.clj:108)
	at zookeeper.internal$make_watcher$reify__440.process(internal.clj:56)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
INFO [2022-03-13 03:19:28,261] jepsen node n2-EventThread - org.apache.zookeeper.ClientCnxn EventThread shut down
INFO [2022-03-13 03:19:28,266] jepsen worker 3 - jepsen.util 3	:ok	:read	0
INFO [2022-03-13 03:19:28,276] jepsen worker 4-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n5.jepsen_jepsen/172.19.0.2:2181, initiating session
INFO [2022-03-13 03:19:28,326] jepsen worker 0 - jepsen.util 5	:ok	:read	0
INFO [2022-03-13 03:19:28,908] jepsen worker 1-SendThread(jepsen-n2.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n2.jepsen_jepsen/172.19.0.6:2181
INFO [2022-03-13 03:19:28,910] jepsen worker 1-SendThread(jepsen-n2.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n2.jepsen_jepsen/172.19.0.6:2181, initiating session
INFO [2022-03-13 03:19:28,918] jepsen worker 1-SendThread(jepsen-n2.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x10143a37d220005 has expired, closing socket connection
ERROR [2022-03-13 03:19:29,023] jepsen worker 1-EventThread - org.apache.zookeeper.ClientCnxn Error while calling watcher
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /stm/atoms/jepsen
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
	at zookeeper$exists.invokeStatic(zookeeper.clj:96)
	at zookeeper$exists.doInvoke(zookeeper.clj:54)
	at clojure.lang.RestFn.invoke(RestFn.java:464)
	at avout.atoms.DistributedAtom.invalidateCache(atoms.clj:107)
	at avout.atoms.DistributedAtom$fn__1307.invoke(atoms.clj:108)
	at zookeeper.internal$make_watcher$reify__440.process(internal.clj:56)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
ERROR [2022-03-13 03:19:29,027] jepsen worker 1-EventThread - org.apache.zookeeper.ClientCnxn Error while calling watcher
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /stm/atoms/jepsen
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
	at zookeeper$exists.invokeStatic(zookeeper.clj:96)
	at zookeeper$exists.doInvoke(zookeeper.clj:54)
	at clojure.lang.RestFn.invoke(RestFn.java:464)
	at avout.atoms.DistributedAtom.invalidateCache(atoms.clj:107)
	at avout.atoms.DistributedAtom$fn__1307.invoke(atoms.clj:108)
	at zookeeper.internal$make_watcher$reify__440.process(internal.clj:56)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
INFO [2022-03-13 03:19:29,028] jepsen worker 1-EventThread - org.apache.zookeeper.ClientCnxn EventThread shut down
INFO [2022-03-13 03:19:29,543] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n5.jepsen_jepsen/172.19.0.2:2181
INFO [2022-03-13 03:19:29,546] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n5.jepsen_jepsen/172.19.0.2:2181, initiating session
INFO [2022-03-13 03:19:29,565] jepsen node n5-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x40143b19ff50000 has expired, closing socket connection
ERROR [2022-03-13 03:19:29,566] jepsen node n5-EventThread - org.apache.zookeeper.ClientCnxn Error while calling watcher
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /stm/atoms/jepsen
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
	at zookeeper$exists.invokeStatic(zookeeper.clj:96)
	at zookeeper$exists.doInvoke(zookeeper.clj:54)
	at clojure.lang.RestFn.invoke(RestFn.java:464)
	at avout.atoms.DistributedAtom.invalidateCache(atoms.clj:107)
	at avout.atoms.DistributedAtom$fn__1307.invoke(atoms.clj:108)
	at zookeeper.internal$make_watcher$reify__440.process(internal.clj:56)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
INFO [2022-03-13 03:19:29,566] jepsen node n5-EventThread - org.apache.zookeeper.ClientCnxn EventThread shut down
INFO [2022-03-13 03:19:29,589] jepsen worker 4-SendThread(jepsen-n5.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Session establishment complete on server jepsen-n5.jepsen_jepsen/172.19.0.2:2181, sessionid = 0x40143b19ff50002, negotiated timeout = 5000
INFO [2022-03-13 03:19:29,736] jepsen worker 4 - jepsen.util 9	:ok	:write	3
INFO [2022-03-13 03:19:29,887] jepsen worker 0 - jepsen.util 5	:invoke	:cas	[4 2]
INFO [2022-03-13 03:19:29,940] jepsen worker 0 - jepsen.util 5	:fail	:cas	[4 2]
INFO [2022-03-13 03:19:29,940] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Opening socket connection to server jepsen-n4.jepsen_jepsen/172.19.0.5:2181
INFO [2022-03-13 03:19:29,946] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Socket connection established to jepsen-n4.jepsen_jepsen/172.19.0.5:2181, initiating session
INFO [2022-03-13 03:19:29,949] jepsen node n4-SendThread(jepsen-n4.jepsen_jepsen:2181) - org.apache.zookeeper.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x30143b1a0c10000 has expired, closing socket connection
ERROR [2022-03-13 03:19:30,051] jepsen node n4-EventThread - org.apache.zookeeper.ClientCnxn Error while calling watcher
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /stm/atoms/jepsen
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
	at zookeeper$exists.invokeStatic(zookeeper.clj:96)
	at zookeeper$exists.doInvoke(zookeeper.clj:54)
	at clojure.lang.RestFn.invoke(RestFn.java:464)
	at avout.atoms.DistributedAtom.invalidateCache(atoms.clj:107)
	at avout.atoms.DistributedAtom$fn__1307.invoke(atoms.clj:108)
	at zookeeper.internal$make_watcher$reify__440.process(internal.clj:56)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
ERROR [2022-03-13 03:19:30,052] jepsen node n4-EventThread - org.apache.zookeeper.ClientCnxn Error while calling watcher
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /stm/atoms/jepsen
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1003)
	at zookeeper$exists.invokeStatic(zookeeper.clj:96)
	at zookeeper$exists.doInvoke(zookeeper.clj:54)
	at clojure.lang.RestFn.invoke(RestFn.java:464)
	at avout.atoms.DistributedAtom.invalidateCache(atoms.clj:107)
	at avout.atoms.DistributedAtom$fn__1307.invoke(atoms.clj:108)
	at zookeeper.internal$make_watcher$reify__440.process(internal.clj:56)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
INFO [2022-03-13 03:19:30,056] jepsen node n4-EventThread - org.apache.zookeeper.ClientCnxn EventThread shut down
INFO [2022-03-13 03:19:31,466] jepsen worker 4 - org.apache.zookeeper.ZooKeeper Session: 0x40143b19ff50002 closed
INFO [2022-03-13 03:19:31,467] jepsen worker 4-EventThread - org.apache.zookeeper.ClientCnxn EventThread shut down
INFO [2022-03-13 03:19:31,469] jepsen worker 3-EventThread - org.apache.zookeeper.ClientCnxn EventThread shut down
INFO [2022-03-13 03:19:31,471] jepsen worker 3 - org.apache.zookeeper.ZooKeeper Session: 0x30143b1a0c10001 closed
INFO [2022-03-13 03:19:31,487] jepsen worker 2 - org.apache.zookeeper.ZooKeeper Session: 0x20143a37d400006 closed
INFO [2022-03-13 03:19:31,487] jepsen worker 2-EventThread - org.apache.zookeeper.ClientCnxn EventThread shut down
INFO [2022-03-13 03:19:31,492] jepsen worker 0 - org.apache.zookeeper.ZooKeeper Session: 0x143a37d150006 closed
INFO [2022-03-13 03:19:31,492] jepsen worker 0-EventThread - org.apache.zookeeper.ClientCnxn EventThread shut down
INFO [2022-03-13 03:19:31,577] jepsen node n1-EventThread - org.apache.zookeeper.ClientCnxn EventThread shut down
INFO [2022-03-13 03:19:31,580] real-pmap 0 - org.apache.zookeeper.ZooKeeper Session: 0x143a37d150004 closed
INFO [2022-03-13 03:19:31,597] jepsen node n3-EventThread - org.apache.zookeeper.ClientCnxn EventThread shut down
INFO [2022-03-13 03:19:31,603] real-pmap 2 - org.apache.zookeeper.ZooKeeper Session: 0x20143a37d400005 closed
INFO [2022-03-13 03:19:31,748] jepsen test runner - jepsen.core Run complete, writing
INFO [2022-03-13 03:19:31,798] jepsen test runner - jepsen.core Snarfing log files
INFO [2022-03-13 03:19:31,912] jepsen node n1 - jepsen.core downloading /var/log/zookeeper/zookeeper.log to zookeeper.log
INFO [2022-03-13 03:19:31,913] jepsen node n3 - jepsen.core downloading /var/log/zookeeper/zookeeper.log to zookeeper.log
INFO [2022-03-13 03:19:31,920] jepsen node n4 - jepsen.core downloading /var/log/zookeeper/zookeeper.log to zookeeper.log
INFO [2022-03-13 03:19:31,931] jepsen node n2 - jepsen.core downloading /var/log/zookeeper/zookeeper.log to zookeeper.log
INFO [2022-03-13 03:19:31,932] jepsen node n5 - jepsen.core downloading /var/log/zookeeper/zookeeper.log to zookeeper.log
INFO [2022-03-13 03:19:32,321] jepsen test runner - jepsen.core Snarfing log files
INFO [2022-03-13 03:19:32,428] jepsen node n2 - jepsen.core downloading /var/log/zookeeper/zookeeper.log to zookeeper.log
INFO [2022-03-13 03:19:32,437] jepsen node n1 - jepsen.core downloading /var/log/zookeeper/zookeeper.log to zookeeper.log
INFO [2022-03-13 03:19:32,438] jepsen node n3 - jepsen.core downloading /var/log/zookeeper/zookeeper.log to zookeeper.log
INFO [2022-03-13 03:19:32,443] jepsen node n4 - jepsen.core downloading /var/log/zookeeper/zookeeper.log to zookeeper.log
INFO [2022-03-13 03:19:32,460] jepsen node n5 - jepsen.core downloading /var/log/zookeeper/zookeeper.log to zookeeper.log
INFO [2022-03-13 03:19:32,702] jepsen node n1 - jepsen.zookeeper n1 tearing down ZK
INFO [2022-03-13 03:19:32,705] jepsen node n2 - jepsen.zookeeper n2 tearing down ZK
INFO [2022-03-13 03:19:32,709] jepsen node n3 - jepsen.zookeeper n3 tearing down ZK
INFO [2022-03-13 03:19:32,716] jepsen node n4 - jepsen.zookeeper n4 tearing down ZK
INFO [2022-03-13 03:19:32,730] jepsen node n5 - jepsen.zookeeper n5 tearing down ZK
INFO [2022-03-13 03:19:33,046] jepsen test runner - jepsen.core Analyzing...
WARN [2022-03-13 03:19:33,318] clojure-agent-send-off-pool-4 - jepsen.checker Error rendering linearizability analysis
java.lang.AssertionError: Assert failed: Expected index but got nil for op #knossos.op.Op{:process nil, :type nil, :f nil, :value nil, :index nil}
i
	at knossos.linear.report$time_coords$fn__9271.invoke(report.clj:184)
	at clojure.core$map$fn__5851.invoke(core.clj:2753)
	at clojure.lang.LazySeq.sval(LazySeq.java:42)
	at clojure.lang.LazySeq.seq(LazySeq.java:51)
	at clojure.lang.RT.seq(RT.java:531)
	at clojure.core$seq__5387.invokeStatic(core.clj:137)
	at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:24)
	at clojure.core.protocols$fn__8131.invokeStatic(protocols.clj:75)
	at clojure.core.protocols$fn__8131.invoke(protocols.clj:75)
	at clojure.core.protocols$fn__8073$G__8068__8086.invoke(protocols.clj:13)
	at clojure.core$reduce.invokeStatic(core.clj:6828)
	at clojure.core$into.invokeStatic(core.clj:6895)
	at clojure.core$into.invoke(core.clj:6887)
	at knossos.linear.report$time_coords.invokeStatic(report.clj:194)
	at knossos.linear.report$time_coords.invoke(report.clj:177)
	at knossos.linear.report$learnings.invokeStatic(report.clj:442)
	at knossos.linear.report$learnings.invoke(report.clj:422)
	at knossos.linear.report$render_analysis_BANG_.invokeStatic(report.clj:654)
	at knossos.linear.report$render_analysis_BANG_.invoke(report.clj:649)
	at jepsen.checker$linearizable$reify__9498$fn__9500.invoke(checker.clj:207)
	at jepsen.checker$linearizable$reify__9498.check(checker.clj:205)
	at jepsen.checker$check_safe.invokeStatic(checker.clj:81)
	at jepsen.checker$check_safe.invoke(checker.clj:74)
	at jepsen.checker$compose$reify__9463$fn__9465.invoke(checker.clj:97)
	at clojure.core$pmap$fn__8447$fn__8448.invoke(core.clj:7022)
	at clojure.core$binding_conveyor_fn$fn__5739.invoke(core.clj:2030)
	at clojure.lang.AFn.call(AFn.java:18)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
@nurturenature
Copy link
Contributor

Hi Jack,

i modify dockerfile for debian 10

See #532

I was never able to create a Docker config that fully worked on Debian 10 or Ubuntu 20.04. The best I could do was get it to come up without having to power cycle, then would have to manually attach to the cluster and finish configuring the environment, etc. There is reported success with jepsen/master and current Docker running privileged on MacOS.

Using LXC vs Docker has been very productive.

use the Zookeeper Debian package for installation

Similar to #527?

If the db changes its API, semantics, behavior, etc, then like all apps, the client also needs to be updated.

@jack-tang1
Copy link
Author

Hi,nurturenature,
I was create dockerfile on windows 10 or Ubuntu 20.04, it has the same error.Or i can try build a docker config on MacOS

Using LXC vs Docker has been very productive.

thanks for your reply, i'll try it again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nurturenature @jack-tang1