Claude 3.5 Sonnet in AWS Bedrock EU Region: "The provided model doesn't support on-demand throughput"

[gllona]

What happened?

Configured LibreChat (DockerHub version latest of today 11/09/24) running in docker compose with deploy-compose.yml, to use AWS Bedrock with models Claude 3 Haiku and Claude 3.5 Sonnet.

Trying to use Claude 3.5 Sonnet produces the following entry in the LibreChat-API container logs:

2024-09-11 11:50:47 error: [api/server/controllers/agents/client.js #sendCompletion] Unhandled error type The provided model doesn't support on-demand throughput.
2024-09-11 11:50:47 error: [handleAbortError] AI response error; aborting request: The provided model doesn't support on-demand throughput.

My librechat.yaml file:

# For more information, see the Configuration Guide:
# https://docs.librechat.ai/install/configuration/custom_config.html

# Configuration version (required)
version: 1.0.5
# This setting caches the config file for faster loading across app lifecycle
cache: true

endpoints:
  bedrock:
    availableRegions:
      - "eu-west-3"
    # streamRate: 35
    titleModel: 'anthropic.claude-3-haiku-20240307-v1:0'

(I commented out the streamRate setting trying to make it work)

In the UI the following message is displayed:

Something went wrong. Here's the specific error message we encountered: An error occurred while processing your request. Please contact the Admin.

Claude Haiku 3 is working fine.

Thank you for creating this amazing tool, it makes sense not to filter private data out to big models training!

Steps to Reproduce

1. Update LibreChat to the latest docker images as for today
2. Configure settings for AWS Bedrock, including Claude Haiku 3 and Claude 3.5 Sonnet models
3. Restart LibreChat
4. Try to chat with the AWS Bedrock Claude 3.5 Sonnet model
5. Check the error in the UI
6. Check logs with docker logs LibreChat-API

What browsers are you seeing the problem on?

Chrome

Relevant log output

2024-09-11 11:50:47 error: [api/server/controllers/agents/client.js #sendCompletion] Unhandled error type The provided model doesn't support on-demand throughput.
2024-09-11 11:50:47 error: [handleAbortError] AI response error; aborting request: The provided model doesn't support on-demand throughput.

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

I moved this to troubleshooting discussions since it's not reproducible. There may be limitations on the region among one of many possible limitations, or for using this exact model ID in your region.

In this particular case, you may need to make use of the environment variable BEDROCK_AWS_MODELS to specify the model ID because it seems 3.5-sonnet in the EU has a separate ID: eu.anthropic.claude-3-5-sonnet-20240620-v1:0

I experienced the same issue when attempting to use Anthropic Claude 3.5 Sonnet in the Paris region.

The provided model doesn't support on-demand throughput.

It seems that the model can only be used with something called an inference profile due to cross-region inference1.

Anyway, it boils down to simply changing the modelId used, based on the region in question.

E.g. In my case, the modelId was anthropic.claude-3-5-sonnet-20240620-v1:0 and now it's eu.anthropic.claude-3-5-sonnet-20240620-v1:0.

HTH.

Footnotes

1. Never heard of this before today. [↩](#user-content-fnref-1-cf8798d39511bfc643cbed61d819fbda)

source: aws-samples/bedrock-claude-chat#508 (comment)

---

a note on this:

  bedrock:
    availableRegions:
      - "eu-west-3"

You should only configure availableRegions if you are working with more than 1 region. This allows the flexibility of "switching" between regions.

[danny-avila]

I wasn't aware there could be additional prefixes like eu in the model ID until now, so I will be updating the frontend to handle that possibility in case you experience anything unexpected with this model ID.

[gllona]

Thank you @danny-avila , this worked perfectly after the changes you proposed. Thank you also for your fast response.