Skip to content

Commit 4f25862

Browse files
feat(zookeeper): backport ZOOKEEPER-4846, ZOOKEEPER-4921, ZOOKEEPER-4925 into Zookeeper 3.9.3 (#1150)
* feat: backport ZOOKEEPER-4846, ZOOKEEPER-4921, ZOOKEEPER-4925 * chore: changelog --------- Co-authored-by: Nick <[email protected]>
1 parent a6ce924 commit 4f25862

5 files changed

+838
-18
lines changed

CHANGELOG.md

Lines changed: 19 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -43,14 +43,14 @@ All notable changes to this project will be documented in this file.
4343
- nifi: Add [nifi-iceberg-bundle](https://github.com/stackabletech/nifi-iceberg-bundle) for NiFi `2.2.0` ([#1060], [#1106]).
4444
- java: Add JDK 24 ([#1097]).
4545
- ci: Add golang image to mirror workflow ([#1103]).
46-
- omid: bump version to 1.1.3 ([#1105])
47-
- hbase: add 2.6.2 and upgrade dependencies ([#1101])
48-
- kafka: Add `4.0.0` ([#1117])
49-
- Include `.tar.gz` snapshots of the product source code in container images ([#1126])
50-
- airflow: OPA authorizer for Airflow 3.x ([#1127])
51-
- kafka: Add `3.9.1` ([#1149])
52-
- spark-k8s: Add `3.5.6` ([#1142])
53-
- spark-connect-client: Add `3.5.6` ([#1142])
46+
- omid: bump version to 1.1.3 ([#1105]).
47+
- hbase: add 2.6.2 and upgrade dependencies ([#1101]).
48+
- kafka: Add `4.0.0` ([#1117]).
49+
- Include `.tar.gz` snapshots of the product source code in container images ([#1126]).
50+
- airflow: OPA authorizer for Airflow 3.x ([#1127]).
51+
- kafka: Add `3.9.1` ([#1149]).
52+
- spark-k8s: Add `3.5.6` ([#1142]).
53+
- spark-connect-client: Add `3.5.6` ([#1142]).
5454
- git-sync: Bump version to 4.4.1 ([#1151]).
5555

5656
### Changed
@@ -66,16 +66,17 @@ All notable changes to this project will be documented in this file.
6666
- yq: Bump products to use `4.45.2` ([#1090]).
6767
- cyclonedx-bom: Bump airflow and superset to use `6.0.0` ([#1090]).
6868
- vector: Bump to `0.46.1` ([#1098]).
69-
- spark: update dependencies for 3.5.5 ([#1094])
70-
- nifi: include NAR SBOMs ([#1119])
69+
- spark: update dependencies for 3.5.5 ([#1094]).
70+
- nifi: include NAR SBOMs ([#1119]).
7171
- nifi: update patch allowing to bypass host header validation starting with NiFi 2.4.0 ([#1125]).
7272
- BREAKING: kcat: Stop building kcat image ([#1124]).
73-
- containerdebug updated to 0.2.0 ([#1128])
74-
- Build Hadoop as `stackable` and configure the Stackable Nexus build-repo for the `root` user ([#1133])
73+
- containerdebug updated to 0.2.0 ([#1128]).
74+
- Build Hadoop as `stackable` and configure the Stackable Nexus build-repo for the `root` user ([#1133]).
7575
- patchable: The base branch is now configured as the git upstream branch ([#1131]).
76-
- airflow: Updates the entrypoint script and removes the check for GID == 0 ([#1138])
76+
- airflow: Updates the entrypoint script and removes the check for GID == 0 ([#1138]).
7777
- druid: Bump druiod-opa-authorizer to `0.7.0` ([#1139]).
7878
- vector: Bump to `0.47.0` ([#1152]).
79+
- zookeeper: backport ZOOKEEPER-4846, ZOOKEEPER-4921, ZOOKEEPER-4925 into Zookeeper 3.9.3 ([#1150]).
7980

8081
### Fixed
8182

@@ -105,16 +106,16 @@ All notable changes to this project will be documented in this file.
105106
Also remove the old release workflow.
106107
- zookeeper: Remove 3.9.2 ([#1093]).
107108
- Remove ubi8-rust-builder image ([#1091]).
108-
- spark: remove 3.5.2 ([#1094])
109+
- spark: remove 3.5.2 ([#1094]).
109110
- hadoop: Remove `3.3.4` and `3.4.0` ([#1099]).
110111
- opa: Remove `0.67.1` ([#1103]).
111112
- opa: Remove legacy bundle-builder from container build ([#1103]).
112113
- omid: Remove 1.1.3-SNAPSHOT ([#1105]).
113114
- hbase: Remove 2.4.18 ([#1101])
114115
- druid: Remove `30.0.0` ([#1110]).
115116
- nifi: Remove `2.2.0` ([#1114]).
116-
- kafka: Remove `3.7.1` and `3.8.0` ([#1117])
117-
- spark-connect-client: Remove `3.5.5` ([#1142])
117+
- kafka: Remove `3.7.1` and `3.8.0` ([#1117]).
118+
- spark-connect-client: Remove `3.5.5` ([#1142]).
118119

119120
[nifi-iceberg-bundle]: https://github.com/stackabletech/nifi-iceberg-bundle
120121
[#1025]: https://github.com/stackabletech/docker-images/pull/1025
@@ -169,8 +170,9 @@ All notable changes to this project will be documented in this file.
169170
[#1137]: https://github.com/stackabletech/docker-images/pull/1137
170171
[#1138]: https://github.com/stackabletech/docker-images/pull/1138
171172
[#1139]: https://github.com/stackabletech/docker-images/pull/1139
172-
[#1149]: https://github.com/stackabletech/docker-images/pull/1149
173173
[#1142]: https://github.com/stackabletech/docker-images/pull/1142
174+
[#1149]: https://github.com/stackabletech/docker-images/pull/1149
175+
[#1150]: https://github.com/stackabletech/docker-images/pull/1150
174176
[#1151]: https://github.com/stackabletech/docker-images/pull/1151
175177
[#1152]: https://github.com/stackabletech/docker-images/pull/1152
176178

zookeeper/stackable/patches/3.9.3/0001-Add-CycloneDX-plugin.patch

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Subject: Add CycloneDX plugin
88
1 file changed, 6 insertions(+), 1 deletion(-)
99

1010
diff --git a/pom.xml b/pom.xml
11-
index 6ef4011f..07ae7538 100644
11+
index 6ef4011fe..07ae75387 100644
1212
--- a/pom.xml
1313
+++ b/pom.xml
1414
@@ -925,7 +925,7 @@
Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
From 4004002a9ff08a539a94842ea12a2a449274e968 Mon Sep 17 00:00:00 2001
2+
From: =?UTF-8?q?Andor=20Moln=C3=A1r?= <[email protected]>
3+
Date: Tue, 11 Feb 2025 10:43:20 -0600
4+
Subject: ZOOKEEPER-4846: Failure to reload database due to missing ACL
5+
6+
ZOOKEEPER-4846. Fix ACL reference on existing znode when trying to create
7+
Reviewers: cnauroth, eolivelli, ztzg
8+
Author: anmolnar
9+
Closes #2222 from anmolnar/ZOOKEEPER-4846
10+
---
11+
.../org/apache/zookeeper/server/DataTree.java | 5 +++--
12+
.../apache/zookeeper/server/DataTreeTest.java | 16 ++++++++++++++++
13+
2 files changed, 19 insertions(+), 2 deletions(-)
14+
15+
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java b/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java
16+
index 3b61c80d8..af937f834 100644
17+
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java
18+
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/server/DataTree.java
19+
@@ -462,8 +462,9 @@ public class DataTree {
20+
// we did for the global sessions.
21+
Long acls = aclCache.convertAcls(acl);
22+
23+
- Set<String> children = parent.getChildren();
24+
- if (children.contains(childName)) {
25+
+ DataNode existingChild = nodes.get(path);
26+
+ if (existingChild != null) {
27+
+ existingChild.acl = acls;
28+
throw new NodeExistsException();
29+
}
30+
31+
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/server/DataTreeTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/server/DataTreeTest.java
32+
index 07a69f14f..fc20ed320 100644
33+
--- a/zookeeper-server/src/test/java/org/apache/zookeeper/server/DataTreeTest.java
34+
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/server/DataTreeTest.java
35+
@@ -23,6 +23,7 @@ import static org.junit.jupiter.api.Assertions.assertFalse;
36+
import static org.junit.jupiter.api.Assertions.assertNotEquals;
37+
import static org.junit.jupiter.api.Assertions.assertNotNull;
38+
import static org.junit.jupiter.api.Assertions.assertNull;
39+
+import static org.junit.jupiter.api.Assertions.assertThrows;
40+
import static org.junit.jupiter.api.Assertions.assertTrue;
41+
import java.io.ByteArrayInputStream;
42+
import java.io.ByteArrayOutputStream;
43+
@@ -631,6 +632,21 @@ public class DataTreeTest extends ZKTestCase {
44+
}
45+
}
46+
47+
+ @Test
48+
+ public void testCreateNodeFixMissingACL() throws Exception {
49+
+ DataTree dt = new DataTree();
50+
+ ReferenceCountedACLCache aclCache = dt.getReferenceCountedAclCache();
51+
+
52+
+ dt.createNode("/the_parent", new byte[0], ZooDefs.Ids.CREATOR_ALL_ACL, -1, 1, 1, 0);
53+
+ Long aclId = dt.getNode("/the_parent").acl;
54+
+ aclCache.removeUsage(aclId);
55+
+ aclCache.purgeUnused();
56+
+ // try to re-create the parent -> throws NodeExistsException, but fixes the deleted ACL
57+
+ assertThrows(NodeExistsException.class, () ->
58+
+ dt.createNode("/the_parent", new byte[0], ZooDefs.Ids.CREATOR_ALL_ACL, -1, 1, 1, 0));
59+
+ dt.createNode("/the_parent/the_child", new byte[0], ZooDefs.Ids.CREATOR_ALL_ACL, -1, 2, 2, 2);
60+
+ }
61+
+
62+
private DataTree buildDataTreeForTest() {
63+
final DataTree dt = new DataTree();
64+
assertEquals(dt.lastProcessedZxid, 0);
Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
From 90e8e0f44e8a884765b6e7afe8bd779d59136fad Mon Sep 17 00:00:00 2001
2+
From: Kezhu Wang <[email protected]>
3+
Date: Sat, 26 Apr 2025 12:04:01 +0800
4+
Subject: ZOOKEEPER-4921: Retry endlessly to establish a brand-new session
5+
6+
This partially rollback ZOOKEEPER-4508 to keep consistent with versions
7+
prior to 3.9.3 (excluded), so to maintain compatibility with third party
8+
libraries.
9+
10+
Refs: ZOOKEEPER-4508, ZOOKEEPER-4921, ZOOKEEPER-4923 and
11+
https://lists.apache.org/thread/nfb9z7rhgglbjzfxvg4z2m3pks53b3c1
12+
---
13+
.../java/org/apache/zookeeper/ClientCnxn.java | 2 +-
14+
.../zookeeper/test/SessionTimeoutTest.java | 65 +++++++++++++------
15+
2 files changed, 47 insertions(+), 20 deletions(-)
16+
17+
diff --git a/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java b/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java
18+
index 0bf616c61..207bb8c49 100644
19+
--- a/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java
20+
+++ b/zookeeper-server/src/main/java/org/apache/zookeeper/ClientCnxn.java
21+
@@ -1242,7 +1242,7 @@ public class ClientCnxn {
22+
to = connectTimeout - clientCnxnSocket.getIdleSend();
23+
}
24+
25+
- int expiration = expirationTimeout - clientCnxnSocket.getIdleRecv();
26+
+ int expiration = sessionId == 0 ? Integer.MAX_VALUE : expirationTimeout - clientCnxnSocket.getIdleRecv();
27+
if (expiration <= 0) {
28+
String warnInfo = String.format(
29+
"Client session timed out, have not heard from server in %dms for session id 0x%s",
30+
diff --git a/zookeeper-server/src/test/java/org/apache/zookeeper/test/SessionTimeoutTest.java b/zookeeper-server/src/test/java/org/apache/zookeeper/test/SessionTimeoutTest.java
31+
index 7a59f5eb9..9f5943f68 100644
32+
--- a/zookeeper-server/src/test/java/org/apache/zookeeper/test/SessionTimeoutTest.java
33+
+++ b/zookeeper-server/src/test/java/org/apache/zookeeper/test/SessionTimeoutTest.java
34+
@@ -18,6 +18,9 @@
35+
36+
package org.apache.zookeeper.test;
37+
38+
+import static org.hamcrest.MatcherAssert.assertThat;
39+
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
40+
+import static org.hamcrest.Matchers.lessThan;
41+
import static org.junit.jupiter.api.Assertions.assertNotNull;
42+
import static org.junit.jupiter.api.Assertions.assertNull;
43+
import static org.junit.jupiter.api.Assertions.assertThrows;
44+
@@ -31,12 +34,15 @@ import java.util.List;
45+
import java.util.concurrent.CompletableFuture;
46+
import java.util.concurrent.CountDownLatch;
47+
import java.util.concurrent.TimeUnit;
48+
+import java.util.concurrent.TimeoutException;
49+
import org.apache.zookeeper.CreateMode;
50+
import org.apache.zookeeper.KeeperException;
51+
import org.apache.zookeeper.TestableZooKeeper;
52+
import org.apache.zookeeper.WatchedEvent;
53+
import org.apache.zookeeper.Watcher;
54+
import org.apache.zookeeper.ZooDefs;
55+
+import org.apache.zookeeper.ZooKeeper;
56+
+import org.apache.zookeeper.common.Time;
57+
import org.junit.jupiter.api.BeforeEach;
58+
import org.junit.jupiter.api.Test;
59+
import org.slf4j.Logger;
60+
@@ -54,6 +60,21 @@ public class SessionTimeoutTest extends ClientBase {
61+
zk = createClient();
62+
}
63+
64+
+ private static class ExpiredWatcher implements Watcher {
65+
+ public volatile CompletableFuture<Void> expired = new CompletableFuture<>();
66+
+
67+
+ synchronized void reset() {
68+
+ expired = new CompletableFuture<>();
69+
+ }
70+
+
71+
+ @Override
72+
+ public synchronized void process(WatchedEvent event) {
73+
+ if (event.getState() == Event.KeeperState.Expired) {
74+
+ expired.complete(null);
75+
+ }
76+
+ }
77+
+ }
78+
+
79+
private static class BusyServer implements AutoCloseable {
80+
private final ServerSocket server;
81+
private final Socket client;
82+
@@ -143,17 +164,24 @@ public class SessionTimeoutTest extends ClientBase {
83+
// stop client also to gain less distraction
84+
zk.close();
85+
86+
- // small connection timeout to gain quick ci feedback
87+
- int sessionTimeout = 3000;
88+
- CompletableFuture<Void> expired = new CompletableFuture<>();
89+
+ // given: established session
90+
+ int sessionTimeout = 3000; // small connection timeout to gain quick ci feedback
91+
+ ExpiredWatcher watcher = new ExpiredWatcher();
92+
zk = createClient(new CountdownWatcher(), hostPort, sessionTimeout);
93+
- zk.register(event -> {
94+
- if (event.getState() == Watcher.Event.KeeperState.Expired) {
95+
- expired.complete(null);
96+
- }
97+
- });
98+
+ zk.register(watcher);
99+
+
100+
+ // when: all server down
101+
+ long start = Time.currentElapsedTime();
102+
+ zk.sync("/"); // touch timeout counts
103+
stopServer();
104+
- expired.join();
105+
+
106+
+ // then: get Expired after session timeout
107+
+ watcher.expired.join();
108+
+ long elapsed = Time.currentElapsedTime() - start;
109+
+ assertThat(elapsed, greaterThanOrEqualTo((long) zk.getSessionTimeout()));
110+
+ assertThat(elapsed, lessThan(zk.getSessionTimeout() * 10L));
111+
+
112+
+ // then: future request will get SessionExpiredException
113+
assertThrows(KeeperException.SessionExpiredException.class, () -> zk.exists("/", null));
114+
}
115+
116+
@@ -162,18 +190,17 @@ public class SessionTimeoutTest extends ClientBase {
117+
// stop client also to gain less distraction
118+
zk.close();
119+
120+
+ // given: unavailable cluster
121+
stopServer();
122+
123+
- // small connection timeout to gain quick ci feedback
124+
- int sessionTimeout = 3000;
125+
- CompletableFuture<Void> expired = new CompletableFuture<>();
126+
- new TestableZooKeeper(hostPort, sessionTimeout, event -> {
127+
- if (event.getState() == Watcher.Event.KeeperState.Expired) {
128+
- expired.complete(null);
129+
- }
130+
- });
131+
- expired.join();
132+
- assertThrows(KeeperException.SessionExpiredException.class, () -> zk.exists("/", null));
133+
+ // when: try to establish a brand-new session
134+
+ int sessionTimeout = 300; // small connection timeout to gain quick ci feedback
135+
+ ExpiredWatcher watcher = new ExpiredWatcher();
136+
+ try (ZooKeeper zk = new ZooKeeper(hostPort, sessionTimeout, watcher)) {
137+
+ // then: never Expired
138+
+ assertThrows(TimeoutException.class, () -> watcher.expired.get(3 * sessionTimeout, TimeUnit.MILLISECONDS));
139+
+ assertThrows(KeeperException.ConnectionLossException.class, () -> zk.exists("/", null));
140+
+ }
141+
}
142+
143+
@Test

0 commit comments

Comments
 (0)