Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ReadTimeout - Exception: Max retries (12) on order ("") hit, raising. #118

Open
metaperl opened this issue May 31, 2018 · 3 comments
Open

ReadTimeout - Exception: Max retries (12) on order ("") hit, raising. #118

metaperl opened this issue May 31, 2018 · 3 comments

Comments

@metaperl
Copy link

I encountered a read timeout after 12 retries. It appears that settings.py does not allow one to configure the number of retries or the delay between them.

How should we move forward with making the code robust in the face of errors such as this transcript shows:

2018-05-31 10:28:06,481 - INFO - ws_thread - Connecting to wss://www.bitmex.com/realtime?subscribe=quote:XBTUSD,trade:XBTUSD,instrument,order:XBTUSD,execution:XBTUSD,margin,position
2018-05-31 10:28:06,482 - INFO - ws_thread - Authenticating with API Key.
2018-05-31 10:28:06,484 - INFO - ws_thread - Started thread
2018-05-31 10:28:07,485 - INFO - ws_thread - Connected to WS. Waiting for data images, this may take a moment...
2018-05-31 10:28:07,486 - INFO - ws_thread - Got all market data. Starting.
2018-05-31 10:28:07,486 - INFO - market_maker - Using symbol XBTUSD.
2018-05-31 10:28:07,486 - INFO - market_maker - Order Manager initializing, connecting to BitMEX. Live run: executing real trades.
2018-05-31 10:28:07,486 - INFO - market_maker - Resetting current position. Canceling all existing orders.
2018-05-31 10:28:07,486 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:28:14,805 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:28:34,821 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:28:42,928 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:29:02,949 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:29:11,774 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:29:31,795 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:29:39,131 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:29:59,149 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:30:09,538 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:30:29,546 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:30:36,897 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:30:56,917 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:31:05,007 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:31:25,028 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:31:34,666 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:31:54,687 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:32:02,063 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:32:22,083 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:32:30,221 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:32:50,242 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:32:57,677 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:33:17,698 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:33:25,768 - WARNING - bitmex - Timed out on request: order (""), retrying...
2018-05-31 10:33:45,789 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:33:53,137 - WARNING - bitmex - Timed out on request: order (""), retrying...
Importing symbol settings for XBTUSD...
Unable to find settings-XBTUSD.py.
Traceback (most recent call last):
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 386, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 382, in _make_request
    httplib_response = conn.getresponse()
  File "/home/schemelab/install/miniconda3/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/home/schemelab/install/miniconda3/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/home/schemelab/install/miniconda3/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/home/schemelab/install/miniconda3/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 291, in recv_into
    raise timeout('The read operation timed out')
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/util/retry.py", line 347, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 388, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 308, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='www.bitmex.com', port=443): Read timed out. (read timeout=7)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 264, in _curl_bitmex
    response = self.session.send(prepped, timeout=timeout)
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/home/schemelab/install/miniconda3/lib/python3.6/site-packages/requests/adapters.py", line 499, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='www.bitmex.com', port=443): Read timed out. (read timeout=7)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./marketmaker", line 4, in <module>
    market_maker.run()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/market_maker.py", line 543, in run
    om = OrderManager()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/market_maker.py", line 220, in __init__
    self.reset()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/market_maker.py", line 223, in reset
    self.exchange.cancel_all_orders()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/market_maker.py", line 60, in cancel_all_orders
    orders = self.bitmex.http_open_orders()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 104, in wrapped
    return fn(self, *args, **kwargs)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 195, in http_open_orders
    verb="GET"
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 256, in retry
    return self._curl_bitmex(path, query, postdict, timeout, verb, rethrow_errors, max_retries)
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 354, in _curl_bitmex
    return retry()
  File "/home/schemelab/prg/sample-market-maker-my-fork/market_maker/bitmex.py", line 252, in retry
    raise Exception("Max retries (%d) on %s (%s) hit, raising." % (max_retries, path, json.dumps(postdict or '')))
Exception: Max retries (12) on order ("") hit, raising.
2018-05-31 10:33:53,147 - INFO - market_maker - Shutting down. All open orders will be cancelled.
2018-05-31 10:33:53,147 - INFO - market_maker - Resetting current position. Canceling all existing orders.
2018-05-31 10:33:53,147 - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order: {"filter": "{\"ordStatus.isTerminated\": false, \"symbol\": \"XBTUSD\"}", "count": 500}
2018-05-31 10:34:00,429 - ERROR - bitmex - Unhandled Error: 400 Client Error: Bad Request for url: https://www.bitmex.com/api/v1/order?filter=%7B%22ordStatus.isTerminated%22%3A+false%2C+%22symbol%22%3A+%22XBTUSD%22%7D&count=500: {"error":{"message":"This request has expired - `expires` is in the past. Current time: 1527777240","name":"HTTPError"}}
2018-05-31 10:34:00,429 - ERROR - bitmex - Endpoint was: GET order: null
@ryanfox
Copy link

ryanfox commented May 31, 2018

You can configure the timeout in settings.py. It defaults to

TIMEOUT = 7

@metaperl
Copy link
Author

I have supplied a patch to make the timeout configurable as well as the number of retries and the delay between them - #119

@ryanfox
Copy link

ryanfox commented Jun 1, 2018

The timeout is already configurable. Also, if the client can't make a successful connection in almost a minute and a half, that suggests issues with your network connection.

Based on your logs, you're trying to trade on the live platform. Keep in mind this from the README:

Develop on Testnet first! Testnet trading is completely free and is identical to the live market.

BitMEX is not responsible for any losses incurred when using this code. This code is intended for sample purposes ONLY - do not use this code for real trades unless you fully understand what it does and what its caveats are.
This is not a sophisticated market making program. It is intended to show the basics of market making while abstracting some of the rote work of interacting with the BitMEX API. It does not make smart decisions and will likely lose money.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@metaperl @ryanfox