Feat/anthropic extended ttl #6205
Open
+512
−23
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added to `cacheBehaviorSchema` extra optional parameters. ``` useExtendedCacheTtlBeta: z.boolean().optional(), cacheTtl: z.enum(["5m", "1h"]).optional(), ```
👷 Deploy request for continuedev pending review.Visit the deploys page to approve it
|
dosubot
bot
added
the
size:L
|
All contributors have signed the CLA ✍️ ✅ |
![recurseml[bot]](https://avatars.githubusercontent.com/in/1054195?s=60&v=4){kind=small}
|
😱 Found 3 issues. Time to roll up your sleeves! 😱 |
|
I have read the CLA Document and I hereby sign the CLA |
|
Some details about how long session with big context looks with 5min cache and 1h cache and cost perspective: |
|
1h ttl |
dosubot
bot
added
size:XL
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Implements granular per-message-type caching for Anthropic models to improve token efficiency in Agent mode. Adds new
CacheBehavioroptions to specify how many of each message type to cache (user messages, tool results, assistant tool calls, etc.) instead of only caching the last 2 user messages.This is related to issue #6135
Checklist
Screenshots
N/A - Backend caching enhancement with no visual changes.
Tests
Added comprehensive test suite
core/llm/llms/Anthropic.enhanced-caching.test.tswith 6 test cases covering:shouldCacheMessagelogicAll tests pass and validate the new per-type caching functionality while maintaining backward compatibility.