Silence errors on Zookeeper shutdown race.

When async shutdowns are used there can be a race between the shutdown
and incoming new watcher events which will produce errors in logs. Silencing
them to avoid unnecessary noise that suggest a problem where none exists.

CHANGELOG.md

@@ -14,6 +14,9 @@ and what APIs have changed, if applicable.
 
 ## [Unreleased]
 
+## [29.33.7] - 2022-05-04
+- Silence Zookeeper errors in logs on race condition between watched events and async shutdown.
+
 ## [29.33.6] - 2022-05-03
 - Provide a mechanism to set a routing hint for the d2 request to get request symbol table.
 
@@ -5230,7 +5233,8 @@ patch operations can re-use these classes for generating patch messages.
 
 ## [0.14.1]
 
-[Unreleased]: https://github.com/linkedin/rest.li/compare/v29.33.6...master
+[Unreleased]: https://github.com/linkedin/rest.li/compare/v29.33.7...master
+[29.33.7]: https://github.com/linkedin/rest.li/compare/v29.33.6...v29.33.7
 [29.33.6]: https://github.com/linkedin/rest.li/compare/v29.33.5...v29.33.6
 [29.33.5]: https://github.com/linkedin/rest.li/compare/v29.33.4...v29.33.5
 [29.33.4]: https://github.com/linkedin/rest.li/compare/v29.33.3...v29.33.4

d2/src/main/java/com/linkedin/d2/discovery/stores/zk/ZKConnection.java

@@ -826,7 +826,21 @@ private class DefaultWatcher implements Watcher
     @Override
     public void process(WatchedEvent watchedEvent)
     {
-      ZooKeeper zk = zk();
+      ZooKeeper zk;
+      try
+      {
+        zk = zk();
+      }
+      catch (IllegalStateException e)
+      {
+        if (_shutdownAsynchronously) {
+          // On asynchronous shutdown, this can be a legitimate race.
+          LOG.debug("Watched event received after connection shutdown (type {}, state {}.", watchedEvent.getType(),
+              watchedEvent.getState());
+          return;
+        }
+        throw e;
+      }
       long sessionID = zk.getSessionId();
 
       if (watchedEvent.getType() == Event.EventType.None)