implement expontential backoff and retry for 'HttpError 500 ... Internal Error'

[cwseys]

Hello,
I experience HttpError 500 ... Internal Error backtraces when using duplicity.
The backtrace (below) indicates that the exception comes from google-api-python-client .

While duplicity might be able to work around the problem in its code, if possible it would be better to fix the problem in google-api-python-client (so that every api user does not have to make their own implementation).

Nivco (apparently of "Google Developer Relations) on stackoverflow suggested implementing an exponential backoff and retry for HTTP Error 500 for others with the same problem:
http://stackoverflow.com/a/12640475

I see a fork which may be of interest called 'implement exponential backoff and retry for BatchHttpRequest':
jay0lee@60eb5f4

Thanks for considering,
C.

[cwseys]

Backtrace:

Attempt 1 failed. HttpError: <HttpError 500 when requesting https://www.googleapis.com/drive/v2/files?q=%270B-PLx8xVPvcDS1N1eFQ5QlYwbVk%27+in+parents+and+trashed%3Dfalse&fields=items%28title%2Cid%29%2CnextPageToken&alt=json&maxResults=1000&pageToken=EAIanwELEgBSlgEKhAEKXPjz6gX_____wxSDokj8DkAA_wH__kNvc21vLnVzZXIoMDAwMDAwY2E5YzRjODRkMlUpLmRpcl9lbnRyeSg4NzAyMDU2NTQyMjZcLjdcLlJFYzdjZGNfTThRKQABEOgHISRTjeXk3XctOQAAAAD6FQwASApQAFoLCSjx1l1vCHwLEAEgATIGCIHrtLAF8ISVfAEMQAAiCwnShEycygAAACAG returned "Internal Error">
Backtrace of previous error: Traceback (innermost last):
File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 365, in inner_retry
return fn(self, _args)
File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 564, in list
return [tobytes(x) for x in self.backend._list()]
File "/usr/lib/python2.7/dist-packages/duplicity/backends/pydrivebackend.py", line 143, in _list
'fields': 'items(title,id),nextPageToken'}).GetList()
File "/usr/local/lib/python2.7/dist-packages/pydrive/apiattr.py", line 154, in GetList
for x in self:
File "/usr/local/lib/python2.7/dist-packages/pydrive/apiattr.py", line 138, in next
result = self._GetList()
File "/usr/local/lib/python2.7/dist-packages/pydrive/auth.py", line 54, in _decorated
return decoratee(self, *args, _kwargs)
File "/usr/local/lib/python2.7/dist-packages/pydrive/files.py", line 56, in _GetList
self.metadata = self.auth.service.files().list(_dict(self)).execute()
File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 142, in positional_wrapper
return wrapped(_args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 729, in execute
raise HttpError(resp, content, uri=self.uri)
HttpError: <HttpError 500 when requesting https://www.googleapis.com/drive/v2/files?q=%270B-PLx8xVPvcDS1N1eFQ5QlYwbVk%27+in+parents+and+trashed%3Dfalse&fields=items%28title%2Cid%29%2CnextPageToken&alt=json&maxResults=1000&pageToken=EAIanwELEgBSlgEKhAEKXPjz6ZL_____wxSCIAvxt4AA_wH__kNvc21vLnVzZXIoMDAwMDAwY2E5YzRjODRkMlUpLmRpcl9lbnRyeSg4NzAyMDU2NTQyMjZcLjdcLlJFUzF1QWR2NkRrKQABENAPISRTjeXk3XctOQAAAABtFgwASBlQAFoLCSjx1l1vCHwLEAEgATIGCLfrtLAF8ISVfAEMQAAiCwnShEycygAAACAG returned "Internal Error">

Giving up after 2 attempts. HttpError: <HttpError 500 when requesting https://www.googleapis.com/drive/v2/files?q=%270B-PLx8xVPvcDS1N1eFQ5QlYwbVk%27+in+parents+and+trashed%3Dfalse&fields=items%28title%2Cid%29%2CnextPageToken&alt=json&maxResults=1000&pageToken=EAIanwELEgBSlgEKhAEKXPjz6ZL_____wxSCIAvxt4AA_wH__kNvc21vLnVzZXIoMDAwMDAwY2E5YzRjODRkMlUpLmRpcl9lbnRyeSg4NzAyMDU2NTQyMjZcLjdcLlJFUzF1QWR2NkRrKQABENAPISRTjeXk3XctOQAAAABtFgwASBlQAFoLCSjx1l1vCHwLEAEgATIGCLfrtLAF8ISVfAEMQAAiCwnShEycygAAACAG returned "Internal Error">
Releasing lockfile <LinkLockFile: '/root/.cache/duplicity/90d3651b4aa73abc6c07294d68123219/tardis01.277403715235550991278502' -- '/root/.cache/duplicity/90d3651b4aa73abc6c07294d68123219/lockfile'>
Removing still remembered temporary file /tmp/duplicity-ZP1AOI-tempdir/mkstemp-wvnVvO-1

[eesheesh]

I think that pull request #160 fixed this.

[cwseys]

I'll test eventually. Isn't it about time for a version bump and new release ?

[nathanielmanistaatgoogle]

@lickdragon: I suspect it's long past time. We'll try to get to it soon.

[thehesiod]

btw, this also applies to batch calls, it gets slightly more complicated there as I believe and sub-request can throw a 5xx. (update) huh, looks like retry logic is mostly there, just needs support for 5xxs: https://github.com/google/google-api-python-client/blob/master/googleapiclient/http.py#L1430 I'll come up with a patch as we're seeing this on our systems. PR: #486

[thehesiod]

btw I think retry for regular execute is now implemented with num_retries parameter

[thehesiod]

ugh, this ALSO applies to oauth, I've seen a 502 on token refresh

[mcdonc]

Related: https://developers.google.com/api-client-library/python/guide/media_upload#resumable-media-chunked-upload

[mcdonc]

Given that #486 was rejected (implement retry for batch requests), and #160 was merged which implements this for regular requests, I don't think there is much worth in keeping this issue open. I'll close it; if you feel this is in error, please let me know.