Anthropic: Make number of attempts configurable

[ThomasBaruzier]

What happened?

Hello, around 5-10% of the time, especially with sonnet in my experience, I encounter multiple stops and restarts while generating one answer.

Example with 2 restarts at line 5 and the last line:

As of 2023, some of the fastest and most cost-effective ARM-based mini computers similar to Raspberry Pi include:

1. Raspberry Pi 4 Model B: While not the newest, it's still one of the most popular and well-supported options.

2. Rock Pi 4: Offers performance comparable to or better than Raspberry Pi 4, with options for more RAM.

3. Odroid N2+As of 2023, some of the fastest low-cost ARM-based mini computers similar to Raspberry Pi include:

1. Raspberry Pi 4 Model B: While not the newest, it's still a popular and powerful option with up to 8GB RAM.

2. Rock Pi 4: Offers performance similar to Raspberry Pi 4 but with some models featuring better GPU performance.

3. Odroid-N2+: Known for its strong CPU performance and efficient cooling.

4. Khadas VIM3: Features a powerful Amlogic A311D SoC with NPU for AI applications.

5. NVIDIA Jetson Nano: While more expensive, it offers excellent GPU performance for its price.

6. Banana Pi M5: Uses the Amlogic S905X3 quad-core ARM Cortex-A55 processor.

7. Orange Pi 5: Features a Rockchip RK3588S octa-core processor, which is very powerful for its class.

8. Radxa ROCK 5B: One of the most powerful in this category, with an octa-core processor and up to 16GB RAM.

The "fastest" can depend on specific use cases and benchmarks. Additionally, availability and prices can vary based on your location and current market conditions. It's worthAs of 2023, some of
---
Error connecting to server, try refreshing the page.

This is caused by a 'overloaded' error from Anthropic, but I don't like that my queries cost 2-3x the price of what they should be when this errors occurs. I'd prefer it to stop and be able to manually continue the unfinished answer.

Steps to Reproduce

Use Anthropic API and be unlucky

What browsers are you seeing the problem on?

Firefox, Chrome, Mobile (Android)

Relevant log output

LibreChat logs:

{"level":"error","message":"[handleAbortError] AI response error; aborting request: Operation failed after 3 attempts: Connection error.","stack":"Error: Operation failed after 3 attempts: Connection error.\n    at AnthropicClient.processResponse (/app/api/app/clients/AnthropicClient.js:811:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async AnthropicClient.sendCompletion (/app/api/app/clients/AnthropicClient.js:824:5)\n    at async AnthropicClient.sendMessage (/app/api/app/clients/BaseClient.js:574:24)\n    at async AskController (/app/api/server/controllers/AskController.js:130:20)\n    at async /app/api/server/routes/ask/anthropic.js:23:5"}
{"code":"ERR_HTTP_HEADERS_SENT","level":"error","message":"[handleAbortError] error while trying to abort message Cannot set headers after they are sent to the client","stack":"Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client\n    at ServerResponse.setHeader (node:_http_outgoing:659:11)\n    at ServerResponse.header (/app/node_modules/express/lib/response.js:794:10)\n    at ServerResponse.send (/app/node_modules/express/lib/response.js:174:12)\n    at ServerResponse.json (/app/node_modules/express/lib/response.js:278:15)\n    at ServerResponse.send (/app/node_modules/express/lib/response.js:162:21)\n    at abortMessage (/app/api/server/middleware/abortMiddleware.js:30:28)\n    at handleAbortError (/app/api/server/middleware/abortMiddleware.js:213:20)\n    at AskController (/app/api/server/controllers/AskController.js:175:5)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /app/api/server/routes/ask/anthropic.js:23:5"}
{"level":"error","message":"[handleAbortError] AI response error; aborting request: Operation failed after 3 attempts: Connection error.","stack":"Error: Operation failed after 3 attempts: Connection error.\n    at AnthropicClient.processResponse (/app/api/app/clients/AnthropicClient.js:811:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async AnthropicClient.sendCompletion (/app/api/app/clients/AnthropicClient.js:824:5)\n    at async AnthropicClient.sendMessage (/app/api/app/clients/BaseClient.js:574:24)\n    at async AskController (/app/api/server/controllers/AskController.js:130:20)\n    at async /app/api/server/routes/ask/anthropic.js:23:5"}
{"code":"ERR_HTTP_HEADERS_SENT","level":"error","message":"[handleAbortError] error while trying to abort message Cannot set headers after they are sent to the client","stack":"Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client\n    at ServerResponse.setHeader (node:_http_outgoing:659:11)\n    at ServerResponse.header (/app/node_modules/express/lib/response.js:794:10)\n    at ServerResponse.send (/app/node_modules/express/lib/response.js:174:12)\n    at ServerResponse.json (/app/node_modules/express/lib/response.js:278:15)\n    at ServerResponse.send (/app/node_modules/express/lib/response.js:162:21)\n    at abortMessage (/app/api/server/middleware/abortMiddleware.js:30:28)\n    at handleAbortError (/app/api/server/middleware/abortMiddleware.js:213:20)\n    at AskController (/app/api/server/controllers/AskController.js:175:5)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /app/api/server/routes/ask/anthropic.js:23:5"}

Screenshots

Anthropic logs:

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

Overloaded issue is on Anthropic’s side. The LibreChat API is intentionally designed to retry up to 3 times.

Not a bug but this is better categorized as a feature request to make the retries configurable.

[ThomasBaruzier]

But it's still a bug
The API shouldn't restart from the beginning of the query on each retry if text has already been generated. Please look at the example I gave.

Also, being able to edit the number of retries seems like a great feature request, as you suggested. It would allow the user to gain finer control over the requests and its expanses.

[danny-avila]

I see, do you mean the fact that the generation cannot be continued even though there was some generation? The overloaded issue is totally independent of LibreChat, and the only way to replicate it would be to send many requests at once.

[ThomasBaruzier]

Kind of

Here's a more clear example:
Certainly! Here are [anthropic error, librechat retries]Certainly! Here are the best price to perf[anthropic error, librechat retries]Certainly! Here [etc.]

Without comments:
Certainly! Here are Certainly! Here are the best price to perfCertainly! Here [etc.]

Ideal:
Certainly! Here are [anthropic error, librechat retries and uses the previous context to continue] the best price to [anthropic error, librechat retries and uses the previous context to continue]performance single board computers [etc.]

Without comments:
Certainly! Here are the best price to performance single board computers [etc.]

---

This bug is mostly annoying with long generations that restart over, expanding the waiting time and the cost further compared to a proper handling of retries "in the middle"

[JonMagon]

Still occurs. Anthropic Request 1 failed: Connection error. in logs.

[TyraVex]

Here's a better example of how the retry mechanism does not take the previous text generated into account: