Auto pacing on 429 error

[exallon72]

It would be great if you could make the API auto pace itself on 429 errors.

[Voyz]

hey @exallon72 thanks for the suggestion 👍

Could you point out how this could be implemented? Would you see IBind keep an internal request counter and throttle requests upon seeing 429? How would you imagine such an interaction would look like from your app's perspective?

[exallon72]

It's a good question. First it should probably not allow any more requests to be done within the same session to that endpoint.

Then I got a strange error. Tried with a couple of BTC. That I apparently do not have access to. It retiried 3 times. But then I got a 429. Indicating to me that it seems to loop a lot more than that since IB is usually not that picky. Unfortunately I don't have the log for the failed attempts left, but should be easy to replicate by using an asset you don't have market data access.

Traceback (most recent call last):
File "/home/jocke/PycharmProjects/test/.venv/lib/python3.12/site-packages/ibind/base/rest_client.py", line 163, in request
return self._process_response(response, result)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jocke/PycharmProjects/test/.venv/lib/python3.12/site-packages/ibind/base/rest_client.py", line 188, in _process_response
raise ExternalBrokerError(f'{self}: response error {result} :: {response.status_code} :: {response.reason} :: {response.text}', status_code=response.status_code) from e
ibind.support.errors.ExternalBrokerError: IbkrClient: response error Result(data=None, request={'url': 'https://192.168.0.13:5000/v1/api/iserver/marketdata/history', 'params': {'conid': '475655817', 'bar': '1min', 'exchange': 'SMART', 'period': '1d', 'outsideRth': False, 'startTime': '20190101-10:00:00'}}) :: 429 :: Too Many Requests :: {"error":"Too many history chart requests, please try again later."}
2024-07-06 08:29:03,492 - ERROR - IbkrClient: response error Result(data=None, request={'url': 'https://192.168.0.13:5000/v1/api/iserver/marketdata/history', 'params': {'conid': '475655817', 'bar': '1min', 'exchange': 'SMART', 'period': '1d', 'outsideRth': False, 'startTime': '20190101-10:00:00'}}) :: 429 :: Too Many Requests :: {"error":"Too many history chart requests, please try again later."}
Traceback (most recent call last):

[exallon72]

One small improvement could be also for the 429 not just send
ExternalBrokerError since you then have to investigate the error to see if it's a 429.

[exallon72]

Adding to that. If you don't have market data access. It should not do any retries. This was using:
marketdata_history_by_conid

[Voyz]

First it should probably not allow any more requests to be done within the same session to that endpoint.

After a 429?

Tried with a couple of BTC. That I apparently do not have access to. It retiried 3 times. But then I got a 429. Indicating to me that it seems to loop a lot more than that since IB is usually not that picky.

What do you mean that 'it seems to loop a lot more'? There's a reattempt logic when encountering ReadTimeout error, but other than that not sure what kind of loop you're referring to here. Could you elaborate?

'https://192.168.0.13:5000/v1/api/iserver/marketdata/history', 'params': {'conid': '475655817', 'bar': '1min', 'exchange': 'SMART', 'period': '1d', 'outsideRth': False, 'startTime': '20190101-10:00:00'}}) :: 429 :: Too Many Requests :: {"error":"Too many history chart requests, please try again later."}

If this response was sent to you after only 3 attempts, I'd encourage you to talk to IBKR support about this and ask as to why it comes up this early.

One small improvement could be also for the 429 not just send ExternalBrokerError since you then have to investigate the error to see if it's a 429.

Sorry, I'm not sure what you're suggesting here. Instead of ExternalBrokerError you'd see some other exception? The ExternalBrokerError has a status_code field that allows us to figure out the cause of the error quickly.

Adding to that. If you don't have market data access. It should not do any retries. This was using: marketdata_history_by_conid

Does IBKR send some particular unique message when we don't have market data access? Then I guess yes, we could totally catch it and handle it gracefully 👍

[exallon72]

What do you mean that 'it seems to loop a lot more'? There's a reattempt logic when encountering ReadTimeout error, but other than that not sure what kind of loop you're referring to here. Could you elaborate?

I am also a bit confused I don't see that only three retries should generate a 429.

You are right about the status_code 429. I am using that now. Thanks for pointing it out.

I will get back to you with log of a request without market data access.