Skip to content

Commit b655852

Browse files
raulcdkou
andauthored
GH-39212: [Release] Retry download binary subprocess in case of OpenSSL error (#39213)
### Rationale for this change In case of OpenSSL error when trying to download RC binaries we do not retry and we are facing some issues with artifactory that after some job retries allowed us to successfully download the binaries. ### What changes are included in this PR? Retry mechanism in case of subprocess error. ### Are these changes tested? Tested locally verifying wheels and jars for 14.0.2 RC 3. ### Are there any user-facing changes? No * Closes: #39212 Lead-authored-by: Raúl Cumplido <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>
1 parent 37eb3b5 commit b655852

File tree

1 file changed

+23
-11
lines changed

1 file changed

+23
-11
lines changed

dev/release/download_rc_binaries.py

+23-11
Original file line numberDiff line numberDiff line change
@@ -121,17 +121,29 @@ def _download_url(self, url, dest_path, *, extra_args=None):
121121
dest_path,
122122
url,
123123
]
124-
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
125-
stderr=subprocess.PIPE)
126-
stdout, stderr = proc.communicate()
127-
if proc.returncode != 0:
128-
try:
129-
# Don't leave possibly partial file around
130-
os.remove(dest_path)
131-
except IOError:
132-
pass
133-
raise Exception(f"Downloading {url} failed\n"
134-
f"stdout: {stdout}\nstderr: {stderr}")
124+
# Retry subprocess in case it fails with OpenSSL Connection errors
125+
# https://issues.apache.org/jira/browse/INFRA-25274
126+
for attempt in range(5):
127+
if attempt > 0:
128+
delay = attempt * 3
129+
print(f"Waiting {delay} seconds before retrying {url}")
130+
time.sleep(delay)
131+
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
132+
stderr=subprocess.PIPE)
133+
stdout, stderr = proc.communicate()
134+
if proc.returncode != 0:
135+
try:
136+
# Don't leave possibly partial file around
137+
os.remove(dest_path)
138+
except IOError:
139+
pass
140+
if "OpenSSL" not in stderr:
141+
# We assume curl has already retried on other errors.
142+
break
143+
else:
144+
return
145+
raise Exception(f"Downloading {url} failed\n"
146+
f"stdout: {stdout}\nstderr: {stderr}")
135147

136148
def _curl_version(self):
137149
cmd = ["curl", "--version"]

0 commit comments

Comments
 (0)