ReadTimeout

[pokidovea]

There is an error while retrieving list of files from folder with almost 100k files. But all files were retrieved successfully.

Traceback (most recent call last):
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    httplib_response = conn.getresponse()
  File ".pyenv/versions/3.6.4/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File ".pyenv/versions/3.6.4/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File ".pyenv/versions/3.6.4/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File ".pyenv/versions/3.6.4/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 291, in recv_into
    raise timeout('The read operation timed out')
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/util/retry.py", line 357, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/connectionpool.py", line 388, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/connectionpool.py", line 308, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.dropboxapi.com', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apps/creatives_manager/processors/base.py", line 144, in _get_files_from_dropbox
    result = self.dropbox.files_list_folder_continue(result.cursor)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/dropbox/base.py", line 1583, in files_list_folder_continue
    None,
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/dropbox/dropbox.py", line 274, in request
    timeout=timeout)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/dropbox/dropbox.py", line 365, in request_json_string_with_retry
    timeout=timeout)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/dropbox/dropbox.py", line 449, in request_json_string
    timeout=timeout,
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/requests/sessions.py", line 559, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/requests/adapters.py", line 526, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.dropboxapi.com', port=443): Read timed out. (read timeout=30)

Repeating of call self.dropbox.files_list_folder_continue(result.cursor) and enlarging timeout don't help.

[greg-db]

Can you share what you tried to increase the timeout?

If you haven't tried it this way, please try this when making the client:

self.dropbox = dropbox.Dropbox(ACCESS_TOKEN, timeout=None)

[pokidovea]

Just endless waiting for a response. May be problem of Dropbox API?

[greg-db]

Thanks for trying that and following up. It sounds like this is due to the API call taking a long time due to the large number of files then.

Please try using a smaller limit, when first calling files_list_folder. The default is effectively in the thousands, so please try something significantly smaller, such as 50.

[pokidovea]

It doesn't help. I've tried 500, 100 and 50 records. The same error at the end

[pokidovea]

Here's my code:

def _get_files_from_dropbox(path: str) -> Generator[List[FileMetadata], None, None]:
    dropbox = Dropbox(settings.DROPBOX_TOKEN)
    try:
        result = dropbox.files_list_folder(path, recursive=True)
    except ReadTimeout:
        # https://github.com/dropbox/dropbox-sdk-python/issues/155
        return [],

    while True:
        files = []
        for entry in result.entries:
            if isinstance(entry, FileMetadata):
                files.append(entry)

        if files:
            yield files

        if not result.has_more:
            break

        try:
            result = dropbox.files_list_folder_continue(result.cursor)
        except ReadTimeout:
            # https://github.com/dropbox/dropbox-sdk-python/issues/155
            return []

[greg-db]

@pokidovea Thanks for trying that and letting me know! I'm afraid I don't have any other workarounds to offer right now, but I'll check in with the team again on this. I'll follow up here when I have any news.

[jtopel]

@greg-db have there been any updates because we're experiencing the same issue as stated above?

[greg-db]

@jtopel No, unfortunately I don't have an update on this.

[jtopel]

@pokidovea I know you have posted your issue to this forum over a year ago. Did you by chance figure out a solution since then?

We are experiencing this same issue right now.

[akalia25]

experiencing the same issue, would love to know if there is a fix

[greg-db]

@akalia25 Thanks for the note! Unfortunately there isn't a simple fix for this, but please refer to my earlier comment for a potential workaround.

[nathanagez]

Hello, we still experience the same issue

[greg-db]

@nathanagez Unfortunately there still isn't a simple fix for this, but please refer to my earlier comment for a potential workaround.

[SamStephens]

Frankly, it's embarassing that this is still open 5 years later.

Either files_list_folder should default to a smaller limit, or the Dropbox client should default to a longer timeout.

We shouldn't be having to manage this ourselves, these are exactly the concerns that a good client library should abstract away.

[greg-db]

@SamStephens Thanks for the feedback! I'll pass this along to the team.