What is the best practice for persisting ChatHistory in Semantic Kernel?

[preezie-yasir]

We are currently using the ChatHistory object from Microsoft.SemanticKernel.ChatCompletion namespace to develop a chat bot like application that can remember your past conversations. We are currently finding it difficult to find the best practice to persist a chat history. My assumption is that sending through your chat history to the GetStreamingChatMessageContentsAsync method consumes tokens for every previous chat/reply from the assistant.

I've heard about the Summarization plugin which can summarize history and we can pass it into the chat history but is this currently the best practice available for our scenario? We are looking to then store the chat history in a persistent database such as Elastic Search for records purposes.

Any help or guidance would be much appreciated.

[sophialagerkranspandey]

In most cases we'd recommend serializing the chat history and store it in Cosmo DB and de-serialize it to use in your application. It's best not to use the summarization plugin as you'll lose fidelity. We've heard this come up several times and will look to create a sample.

[mymaher]

Is it possible if you can please share few lines of code how to perform serializing ChatHistory and saving it in Cosmo DB? As per my experience, ChatHistory cannot be de-serialize.

[dmytrostruk]

@mymaher I'm not sure if we have dedicated example, but we have unit test which verifies that ChatHistory can be serialized/deserialized:

semantic-kernel/dotnet/src/SemanticKernel.UnitTests/AI/ChatCompletion/ChatHistoryTests.cs

Lines 17 to 45 in 382972b

	public void ItCanBeSerializedAndDeserialized()
	{
	// Arrange
	var options = new JsonSerializerOptions();
	ChatHistory chatHistory =
	[
	new ChatMessageContent(AuthorRole.System, "You are a polite bot.") { AuthorName = "ChatBot" },
	new ChatMessageContent(AuthorRole.User, "Hello") { AuthorName = "ChatBot" },
	new ChatMessageContent(AuthorRole.Assistant, "Hi") { AuthorName = "ChatBot" },
	];
	var chatHistoryJson = JsonSerializer.Serialize(chatHistory, options);
	// Act
	var chatHistoryDeserialized = JsonSerializer.Deserialize<ChatHistory>(chatHistoryJson, options);
	// Assert
	Assert.NotNull(chatHistoryDeserialized);
	Assert.Equal(chatHistory.Count, chatHistoryDeserialized.Count);
	for (var i = 0; i < chatHistory.Count; i++)
	{
	Assert.Equal(chatHistory[i].Role.Label, chatHistoryDeserialized[i].Role.Label);
	Assert.Equal(chatHistory[i].Content, chatHistoryDeserialized[i].Content);
	Assert.Equal(chatHistory[i].AuthorName, chatHistoryDeserialized[i].AuthorName);
	Assert.Equal(chatHistory[i].Items.Count, chatHistoryDeserialized[i].Items.Count);
	Assert.Equal(
	chatHistory[i].Items.OfType<TextContent>().Single().Text,
	chatHistoryDeserialized[i].Items.OfType<TextContent>().Single().Text);
	}
	}

[mymaher]

Many thanks @dmytrostruk. It is working perfectly.

[y8l]

@sophialagerkranspandey, @dmytrostruk Thank you for your responses! Instead of serialising the chat history, can we save the chat history into the SemanticTextMemory using TextMemoryPlugin? What are the pros & cons of this approach?
Overall, I want to use planner with plugins to retreive the knowledge (e.g., query the SQL or Azure Search Index), and then save the chat history to SemanticTextMemory, from where I can {recall} the relevant chat history in plugins instead of using the whole {{chat_history}} as a plugin's function parameter. Does it make sense? Is there a better way to make the agent (or planner) always refer to its memory (cache) before calling the plugins?

[cmargraff]

Also do we fetch the whole chat history from cosmos db every time the user hits our api? How would we go about making this less read heavy on the db? Or is that not a possibility right now?

Well there's probably no one good answer here, but token weight growing every turn is just too expensive in a production environment. What I've been doing is subclassing Chat History to stuff a bunch of content (such as session logs) that I'll never send to the completion agent, then filtering the results when I need history for a new kargs. I'm also using a vectorization attribute on my subclassed serialized chat history, adding chat history to a local vector store for intent and context, and only sending non-vectorized items through to the token paywall. It works ok, but I've found I need some maintenance on the system message to keep the agent in tune. I haven't an AI function to do that, but it's coming. I also have some blocking on the vectorization thread to deal with, so mind your async awaits if you do this.

[preezie-yasir]

Tagging these related posts 2247 4770