Tolerate restarts while waiting for job completion

Details: * Requests for job status will return ConnectionError while HMC is restarting. This will be ignored to tolerate HMC restarts that will happen while waiting for completion of a job. Signed-off-by: Abhiram Santhosh <[email protected]>
zhmcclient · Dec 15, 2023 · f0428de · f0428de
1 parent a7300a5
commit f0428de
Show file tree

Hide file tree

Showing 2 changed files with 11 additions and 2 deletions.
diff --git a/docs/changes.rst b/docs/changes.rst
@@ -35,6 +35,10 @@ Released: not yet
 
 **Enhancements:**
 
+* Added support for tolerating HMC restarts while waiting for a job to complete.
+  Session.wait_for_completion() now retries in case of ConnectionError instead of
+  raising the error. (issue #1365)
+
 **Cleanup:**
 
 **Known issues:**

diff --git a/zhmcclient/_session.py b/zhmcclient/_session.py
@@ -1773,7 +1773,6 @@ def wait_for_completion(self, operation_timeout=None):
           :exc:`~zhmcclient.ParseError`
           :exc:`~zhmcclient.ClientAuthError`
           :exc:`~zhmcclient.ServerAuthError`
-          :exc:`~zhmcclient.ConnectionError`
           :exc:`~zhmcclient.OperationTimeout`: The timeout expired while
             waiting for job completion.
         """
@@ -1785,7 +1784,13 @@ def wait_for_completion(self, operation_timeout=None):
             start_time = time.time()
 
         while True:
-            job_status, op_result_obj = self.check_for_completion()
+            try:
+                job_status, op_result_obj = self.check_for_completion()
+            except ConnectionError:
+                HMC_LOGGER.debug("Retrying after ConnectionError while waiting"
+                                 " for completion of job %s. This could be "
+                                 "because HMC is restarting.", self.uri)
+                job_status = None
 
             # We give completion of status priority over strictly achieving
             # the timeout, so we check status first. This may cause a longer