Add support for AWS Bedrock (Nova & Claude) #160

Smiley73 · 2025-01-04T18:51:58Z

No description provided.

valentinfrlch · 2025-01-05T21:21:13Z

Hi, thanks for the PR! Have you tested this yet?

Smiley73 · 2025-01-05T22:26:17Z

Yes I have switched my HA to use this version and it's working great. I'm feeding it 15-30 jpegs extracted from Blue Iris, using nova-pro.
I ran a quick automation to capture the debug output for reference:

2025-01-05 16:21:51.878 DEBUG (MainThread) [custom_components.llmvision.providers] Found model type `us.amazon.nova-pro-v1:0` for AWS Bedrock call.
2025-01-05 16:21:51.879 DEBUG (MainThread) [custom_components.llmvision.providers] AWS Bedrock request data: {'messages': [{'role': 'user', 'content': [{'text': 'Image 1:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 2:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 3:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 4:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 5:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 6:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 7:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 8:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 9:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 10:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 11:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 12:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 13:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 14:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 15:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 16:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 17:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 18:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 19:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 20:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 21:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 22:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 23:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 24:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 25:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 26:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 27:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': 'Image 28:'}, {'image': {'format': 'jpeg', 'source': {'bytes': '<long_string>'}}}, {'text': ''<long_string>'}]}], 'inferenceConfig': {'max_new_tokens': 1000, 'temperature': 0.1}}
2025-01-05 16:21:51.879 INFO (MainThread) [custom_components.llmvision.providers] Invoking Bedrock model us.amazon.nova-pro-v1:0 in us-east-1
2025-01-05 16:21:57.885 DEBUG (MainThread) [custom_components.llmvision.providers] AWS Bedrock call Response: {'ResponseMetadata': {'RequestId': 'f0556879-5c18-4e94-a810-f4daa6054a00', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Sun, 05 Jan 2025 22:21:57 GMT', 'content-type': 'application/json', 'content-length': '468', 'connection': 'keep-alive', 'x-amzn-requestid': 'f0556879-5c18-4e94-a810-f4daa6054a00', 'x-amzn-bedrock-invocation-latency': '5696', 'x-amzn-bedrock-output-token-count': '62', 'x-amzn-bedrock-input-token-count': '19817'}, 'RetryAttempts': 0}, 'contentType': 'application/json', 'body': <botocore.response.StreamingBody object at 0x7f3e71e2b6d0>}
2025-01-05 16:21:57.885 INFO (MainThread) [custom_components.llmvision.providers] AWS Bedrock call latency: 5696 tokens_in: 19817 tokens_out: 62
2025-01-05 16:21:57.885 DEBUG (MainThread) [custom_components.llmvision.providers] AWS Bedrock call response data: {'output': {'message': {'content': [{'text': '{\n  "report": "A person was delivering mail at the front door.",\n  "explanation": "The camera named \'Front Door\' captured a person delivering mail. The person was seen interacting with the mailbox, indicating mail delivery. No other activities, vehicles, or persons were detected in the footage."\n}'}], 'role': 'assistant'}}, 'stopReason': 'end_turn', 'usage': {'inputTokens': 19817, 'outputTokens': 62, 'totalTokens': 19879}}

valentinfrlch · 2025-01-06T08:00:34Z

custom_components/llmvision/providers.py

+
+        return response_text
+
+    async def _post(self, model, data) -> dict:


Provider implementations shouldn't overwrite the _post method. If AWS requires special error handling those errors should be handled directly in Provider.

valentinfrlch · 2025-01-06T08:06:24Z

custom_components/llmvision/providers.py

+            _LOGGER.error(f"Found unknown model type `{call.model}` for AWS Bedrock call.")
+            raise ServiceValidationError("Unknown model type specified. Only Nova and Claude are currently supported.")
+
+    def _prepare_vision_data_nova(self, call) -> list:


This needs to handled in _prepare_vision_data as well as the superclass (Provider) looks like this:

async def vision_request(self, call) -> str: data = self._prepare_vision_data(call) return await self._make_request(data)

Therefore _prepare_vision_data_nova would never be called.

valentinfrlch · 2025-01-06T08:07:52Z

custom_components/llmvision/providers.py

+        else:
+            _LOGGER.warning(f"Found unknown model type `{call.model}` for AWS Bedrock call. Will attempt `Nova`")
+
+    def _prepare_text_data_nova(self, call) -> list:


Also needs to be handled in _prepare_text_data for the same reason as vision above.

valentinfrlch

Amazing work so far! Just a couple of issues (see my comments). Does AWS really have different API guidelines for different models?

Also boto3 would need to be added to requirements in manifest.json see the HA dev docs here.

Thanks again for your work!

Smiley73 · 2025-01-06T12:01:47Z

I'll get it tackled. It'll be a couple days.
Yes Bedrock uses the native formatting for each model. Makes it easier for them to integrate I guess, but more of a pain for anybody wanting flexibility. In that case I guess it would be easier to use something like a lite-llm proxy. I'll dig through their code a bit, to see if I missed anything.

Smiley73 · 2025-01-07T00:38:54Z

@valentinfrlch I made the changes requested. Let me know if I misunderstood anything, my coding skills are quite rusty.
I also checked into the API format. According to their documentation, each model on Bedrock uses its native API format.

youkorr · 2025-01-07T14:46:12Z

Hello, your work is great, but I can't integrate DeepSeek.

Smiley73 · 2025-01-07T15:53:54Z

Hello, your work is great, but I can't integrate DeepSeek.

Can you be a bit more specific? DeepSeek seems to be available through the Marketplace only and you have to provision an instance?
My focus was on Nova and Anthropic through Bedrock directly. Each model has a slightly different API and needs coding.
You might be better of leveraging something like litellm-proxy and then add a custom openai endpoint to llm-vision.

valentinfrlch · 2025-01-07T21:22:06Z

Hello, your work is great, but I can't integrate DeepSeek.

Please create a feature request. This is a pull request for something completely unrelated.

valentinfrlch

Code looks good to me. Thanks again for your contribution! Somehow I don't have access to any models on AWS so I can't really test this... Will check again tomorrow.

Smiley73 · 2025-01-08T23:57:34Z

Code looks good to me. Thanks again for your contribution! Somehow I don't have access to any models on AWS so I can't really test this... Will check again tomorrow.

You need to turn the model access on.
Go to the AWS console -> Bedrock -> Bedrock Configurations (all the way down) -> Model Access
Just turn all of them on. Give some explanation.
If they come out with new versions of the models or new ones, you'll have to do this for anything new.

valentinfrlch · 2025-01-09T12:44:41Z

During validate there is a call to _post. I think it is supposed to go to invoke_bedrock. Sorry, I missed that when reviewing.

https://github.com/Smiley73/ha-llmvision/blob/be80641b7aa2e5b7520b35c54b9b3f7448324706/custom_components/llmvision/providers.py#L892C1-L892C62

Smiley73 · 2025-01-09T14:19:37Z

During validate there is a call to _post. I think it is supposed to go to invoke_bedrock. Sorry, I missed that when reviewing.

https://github.com/Smiley73/ha-llmvision/blob/be80641b7aa2e5b7520b35c54b9b3f7448324706/custom_components/llmvision/providers.py#L892C1-L892C62

Of course the one thing I didn't test! Good catch. I'll fix it tonight.

Smiley73 · 2025-01-10T00:07:51Z

Should be fixed now. Sorry about that.

valentinfrlch · 2025-01-10T08:09:30Z

Thank you! Everything works now. The models seem to be very accurate too, so a great addition!

Smiley73 · 2025-01-13T20:05:15Z

@valentinfrlch I'm going to refactor this. I must have had tunnel vision and missed a different way of invoking the models on Bedrock. Using the converse api it's a single API spec and we'll be able to support the majority of models available in AWS.
https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-call.html#conversation-inference-call-request

valentinfrlch reviewed Jan 6, 2025

View reviewed changes

Smiley73 added 2 commits January 6, 2025 17:43

Add support for AWS Bedrock (Nova & Claude)

de3cf0f

Add Bedrock feature to readme file

f318b7e

Smiley73 force-pushed the aws2 branch from 6b5a1d5 to f318b7e Compare January 6, 2025 23:43

Smiley73 added 3 commits January 6, 2025 17:52

Add boto3 and botocore to the requirements for the component

73f9091

code cleanup base on feedback

c84cf35

fix failed MR check

be80641

Repository owner deleted a comment from youkorr Jan 7, 2025

valentinfrlch approved these changes Jan 8, 2025

View reviewed changes

Repository owner deleted a comment from youkorr Jan 9, 2025

Config validation should use new invoke_bedrock function

17803c2

valentinfrlch merged commit 0f6763f into valentinfrlch:main Jan 10, 2025
4 checks passed

Smiley73 deleted the aws2 branch January 10, 2025 12:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for AWS Bedrock (Nova & Claude) #160

Add support for AWS Bedrock (Nova & Claude) #160

Smiley73 commented Jan 4, 2025

valentinfrlch commented Jan 5, 2025

Smiley73 commented Jan 5, 2025

valentinfrlch Jan 6, 2025

valentinfrlch Jan 6, 2025

valentinfrlch Jan 6, 2025

valentinfrlch left a comment

Smiley73 commented Jan 6, 2025

Smiley73 commented Jan 7, 2025

youkorr commented Jan 7, 2025

Smiley73 commented Jan 7, 2025

valentinfrlch commented Jan 7, 2025

valentinfrlch left a comment

Smiley73 commented Jan 8, 2025

valentinfrlch commented Jan 9, 2025

Smiley73 commented Jan 9, 2025

Smiley73 commented Jan 10, 2025

valentinfrlch commented Jan 10, 2025

Smiley73 commented Jan 13, 2025


		return response_text

		async def _post(self, model, data) -> dict:

Add support for AWS Bedrock (Nova & Claude) #160

Add support for AWS Bedrock (Nova & Claude) #160

Conversation

Smiley73 commented Jan 4, 2025

valentinfrlch commented Jan 5, 2025

Smiley73 commented Jan 5, 2025

valentinfrlch Jan 6, 2025

Choose a reason for hiding this comment

valentinfrlch Jan 6, 2025

Choose a reason for hiding this comment

valentinfrlch Jan 6, 2025

Choose a reason for hiding this comment

valentinfrlch left a comment

Choose a reason for hiding this comment

Smiley73 commented Jan 6, 2025

Smiley73 commented Jan 7, 2025

youkorr commented Jan 7, 2025

Smiley73 commented Jan 7, 2025

valentinfrlch commented Jan 7, 2025

valentinfrlch left a comment

Choose a reason for hiding this comment

Smiley73 commented Jan 8, 2025

valentinfrlch commented Jan 9, 2025

Smiley73 commented Jan 9, 2025

Smiley73 commented Jan 10, 2025

valentinfrlch commented Jan 10, 2025

Smiley73 commented Jan 13, 2025