Improve handling endpoint rate limits

[livthomas]

Prerequisites

• Are you running the latest version(s)?
• Have you searched the documentation for your issue?
• Did you search for an existing issue or pull request?

Description

Having written my own indexer before, I must say that SubQuery is very aggressive with endpoints. I'm using free tiers of 6 different RPC providers which are quite generous in both monthly quota and RPS. But I still hardly see anything expect for rate limit errors in the SubQuery logs. I have also received emails from multiple providers saying that most of my requests were rate limited and I should do something about it. And I only ran this app once for a few minutes. (It is an Ethereum app that uses only event handlers.)

I would very much appreciate if SubQuery respected rate limits. I would propose a three step solution you could gradually implement, starting from the simplest one:

1. Add an option to configure rate limit globally across endpoints.
2. Enable users to specify rate limits per endpoint as @stwiname suggested in Improve multiple endpoints configuration #1925.
3. Detect endpoint rate limits automatically, then respect them in all requests but test it again once in a while.

Details

These details can help to reproduce the environment the issue is occurring

Local Environment: @subql/cli/4.2.5 darwin-arm64 node-v18.18.2
Query Version: @subql/[email protected]
Indexer Version: @subql/[email protected]
Network Details:

• Ethereum Goerli testnet

Steps to Reproduce

1. Use multiple free-tier endpoints.
2. Add a few event handlers.
3. Run the indexer.

Expected behavior: There should be almost no rate limit errors in the logs. SubQuery should know about these limits and respect them.

Actual behavior: Logs are full of rate limit errors. There is barely anything else in there.