Add a way to change the assistant before the reply

[aristofeles]

I've created two assistants for CLAUDE: one with custom instructions (start claude) and one without (default claude). However, it seems I have to reply to a chat before I can switch between them.
Is there a way to change this? What I'm looking for is something similar to the system prompt option in Stable Output, so I don't have to waste tokens adding the instructions to all my replies.

[mattlindsey]

Hi! You can switch right away already and it creates a new 'v2' conversation with the new Assistant, doesn't it? I'm curious because I don't quite understand the issue. Although the way it currently allows to switch between 'versions' does seem a little odd to me.

You want 1 conversation where you can switch back and forth? Or option to turn instructions on/off?

[mattlindsey]

Yeah. The way the conversation currently branches when you do a 'regenerate', you can't add or remove the system message once you start? Is that correct @krschacht?

[krschacht]

Matt, interestingly, when you switch from one assistant to another mid-conversation, all of the message history will be sent to that new assistant and it will actually use the system instructions of the new assistant. When the API call is sent to the LLM it sends the entire message history + the system instructions of whichever Assistant you are asking for a reply from.

[krschacht]

@aristofeles Since you've created two separate Assistants, they should both be listed permanently in your left sidebar. If you want to use your custom instructions, just click the "Start Claude" assistant and then ask it a question. The custom instructions will be included in every conversation you start with it. You don't need to start with "Default Claude" and then switch to "Start Claude."

Does that address your issue?

Also, just for context, you should not need to repeat instructions in every reply. Each time you reply, the entire history of this conversation is included when it communicates with the LLM so any instructions you've already provided will be included again.

[aristofeles]

@krschacht

English is not my first language, so my explanation might be a bit unclear. Let me clarify my issue:

What I`m looking for is a functionality where custom instructions are sent only once at the beginning of the conversation, and not repeated in each interaction with the LLM. Currently, even when the system says 'the entire conversation history is included', this means that the instructions are being resent in each message, unnecessarily consuming tokens.

An ideal solution would be to implement a mechanism similar to Stable Output's 'system prompt', where instructions are set once at the start and kept in memory without needing to be resent in each interaction. This would significantly optimize token usage.

[mattlindsey]

I think that I see. Do you want a way to 'switch' an existing conversation that you started with "Start Claude" to "Default Claude" right away, rather than using the regenerate icon (the two little arrows in a circle) after you have already asked something? Like a switch icon, or ability to change the Assistant in the drop-down above the conversation? That would save a little bit on tokens.

Keep in mind in all cases the LLM itself can't remember system instructions without being sent each time.

[aristofeles]

You are right, my idea was a way to switch not needing to regenerate.

Bit my mistake is in the last part. As I understood, the system instructions (also known as the System Prompt?) should provide guidelines to the LLM. In this context, I thought that giving the instructions at the beginning would be sufficient most of the time.

[krschacht]

@aristofeles You are correct, the instructions only need to be given at the beginning of the conversation. They do not need to be repeated for each message. So when we make an API request to openai our payload looks like this:

config & system instructions
message1
message2
message3

However, the API is stateless so when we want to ask another question, then we re-send the whole history with the next request:

config & system instructions
message1
message2
message3
message4

So there is a sense in which we're re-sending the instructions with each new message, but I don't think it's the sense in which you meant it. However, OpenAI recently introduced an efficiency in their system where they cache the previous request that you made for a conversation and if all of the preceding contents matches then they don't re-process it: https://openai.com/index/api-prompt-caching/

So you shouldn't be charged for re-processing the "instructions through message3" when you submitted message4.

[krschacht]

All that said, if there was some other reason you wanted to switch assistant, let me know. It might be a good thing for us to support regardless, I just haven't run into a good reason for it. But if the only reason you were trying to switch was because of this then we're probably good.