feat: Add cross-region inference support for Bedrock models (#535) #536
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…les#535) This commit introduces the initial setup for supporting cross-region inference in the Bedrock Chat application. The changes include: - Added `is_region_supported_for_inference` function in `utils.py` to check if a region supports inference. - Modified `get_bedrock_client` function in `utils.py` to use cross-region inference if enabled and the region is supported. - Updated `get_model_id` function in `bedrock.py` to use the base model ID for cross-region inference if enabled, the region is supported, and the model is included in `CROSS_REGION_INFERENCE_MODELS`. If any of these conditions are not met, it falls back to using the local model ID and logs a warning. - Added `enableBedrockCrossRegionInference` option to `cdk.json` with the default value set to `false`. These changes lay the foundation for enabling cross-region inference in the Bedrock Chat application. The feature can be enabled or disabled using the `enableBedrockCrossRegionInference` configuration option in `cdk.json`.
…les#535) - Fixing '(' was never closed
…les#535) - Bedrock and utils reformated
2 tasks
statefb
reviewed
Sep 24, 2024
| @@ -40,9 +43,29 @@ def is_running_on_lambda(): | |||
| return "AWS_EXECUTION_ENV" in os.environ | |||
|
|
|||
|
|
|||
| def is_region_supported_for_inference(region: str) -> bool: | |||
There was a problem hiding this comment.
To emphasize for cross region inference, rename it like:
before: is_region_supported_for_inference.
after: is_region_supported_for_cross_inference
| @@ -40,9 +43,29 @@ def is_running_on_lambda(): | |||
| return "AWS_EXECUTION_ENV" in os.environ | |||
|
|
|||
|
|
|||
| def is_region_supported_for_inference(region: str) -> bool: | |||
| supported_regions = [ | |||
There was a problem hiding this comment.
this variable is constant, so please replace with SUPPORTED_REGIONS
| logger.warning( | ||
| f"Cross-region inference is enabled, but the region {region} is not supported. Using default region." | ||
| ) | ||
| return boto3.client("bedrock-runtime", region_name=REGION) |
There was a problem hiding this comment.
Use region i.e. BEDROCK_REGION as default client. REGION is used for services except for bedrock
| @@ -257,26 +265,48 @@ def calculate_price( | |||
| return input_price * input_tokens / 1000.0 + output_price * output_tokens / 1000.0 | |||
|
|
|||
|
|
|||
| CROSS_REGION_INFERENCE_MODELS = { | |||
There was a problem hiding this comment.
move it inside get_model_id since it's used only for the function
| logger.warning( | ||
| f"Cross-region inference is not available for model {model}. Using local model." | ||
| ) | ||
| return f"{base_model_id}-local" |
There was a problem hiding this comment.
Does the postfix i.e. -local work?
| logger.info( | ||
| f"Using cross-region inference for model {model} in region {BEDROCK_REGION}" | ||
| ) | ||
| return base_model_id |
There was a problem hiding this comment.
Need to add prefix e.g. eu, us?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit introduces the initial setup for supporting cross-region inference in the Bedrock Chat application.
Issue #, if available:
#535
Description of changes:
is_region_supported_for_inferencefunction inutils.pyto check if a region supports inference.get_bedrock_clientfunction inutils.pyto use cross-region inference if enabled and the region is supported.get_model_idfunction inbedrock.pyto use the base model ID for cross-region inference if enabled, the region is supported, and the model is included inCROSS_REGION_INFERENCE_MODELS. If any of these conditions are not met, it falls back to using the local model ID and logs a warning.enableBedrockCrossRegionInferenceoption tocdk.jsonwith the default value set tofalse.These changes lay the foundation for enabling cross-region inference in the Bedrock Chat application. The feature can be enabled or disabled using the
enableBedrockCrossRegionInferenceconfiguration option incdk.json.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.