New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for prompt caching (image + video) #149

Open

Blaizzy opened this issue Dec 15, 2024 · 1 comment

Owner

Blaizzy commented Dec 15, 2024

No description provided.

Owner Author

Blaizzy commented Dec 30, 2024 •

edited

Loading

To solve #136
We can load KV cache (attention sink) and compute values for new image and prompt tokens.

Note:

Save each image hash in the cached prompts/features for efficient loading.
Prefix caching (to investigate)

https://docs.vllm.ai/en/latest/automatic_prefix_caching/apc.html

https://www.restack.io/p/large-language-models-answer-prefix-caching-cat-ai

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment