Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: Bedrock batch inference #4

Open
donatoaz opened this issue Dec 10, 2024 · 0 comments
Open

feature: Bedrock batch inference #4

donatoaz opened this issue Dec 10, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@donatoaz
Copy link

As a user I'd like to leverage Bedrock Batch Inference so that batch product onboarding is cost optimized.

Definition of Done (DoD): Batch product onboarding uses Bedrock batch inference

This is challenging because batch inference jobs take the prompts in jsonl (jsonline) format, which would require some major refactoring in the code for our application. However there is a clear potential benefit of slashing costs in half

Bonus: refactoring supports both styles of execution: on demand and batch

@donatoaz donatoaz added the enhancement New feature or request label Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant