Enhancement: Artifacts Instruction Message Placement Affects OpenAI Prompt Caching Efficiency #4327
hongkai-neu
started this conversation in
Feature Requests & Suggestions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
When the Toggle Artifacts UI is enabled, the current version is moving the instruction message to the second-to-last position in the message array. This placement prevents OpenAI from effectively using its prefix matching for caching, as described in their documentation (https://platform.openai.com/docs/guides/prompt-caching).
We should consider modifying our message handling to keep the instruction message in a consistent position, preferably at the beginning of the array, to take full advantage of OpenAI's caching mechanism.
Beta Was this translation helpful? Give feedback.
All reactions