dotnet/notebooks/06-memory-and-embeddings.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building Semantic Memory with Embeddings\n",
    "\n",
    "So far, we've mostly been treating the kernel as a stateless orchestration engine.\n",
    "We send text into a model API and receive text out. \n",
    "\n",
    "In a [previous notebook](04-context-variables-chat.ipynb), we used `context variables` to pass in additional\n",
    "text into prompts to enrich them with more context. This allowed us to create a basic chat experience. \n",
    "\n",
    "However, if you solely relied on context variables, you would quickly realize that eventually your prompt\n",
    "would grow so large that you would run into a the model's token limit. What we need is a way to persist state\n",
    "and build both short-term and long-term memory to empower even more intelligent applications. \n",
    "\n",
    "To do this, we dive into the key concept of `Semantic Memory` in the Semantic Kernel. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "#r \"nuget: Microsoft.SemanticKernel, 0.18.230725.3-preview\"\n",
    "#r \"nuget: System.Linq.Async, 6.0.1\"\n",
    "\n",
    "#!import config/Settings.cs\n",
    "\n",
    "using Microsoft.SemanticKernel;\n",
    "using Microsoft.SemanticKernel.SemanticFunctions;\n",
    "using Microsoft.SemanticKernel.Orchestration;"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to use memory, we need to instantiate the Kernel with a Memory Storage\n",
    "and an Embedding backend. In this example, we make use of the `VolatileMemoryStore`\n",
    "which can be thought of as a temporary in-memory storage (not to be confused with Semantic Memory).\n",
    "\n",
    "This memory is not written to disk and is only available during the app session.\n",
    "\n",
    "When developing your app you will have the option to plug in persistent storage\n",
    "like Azure Cosmos Db, PostgreSQL, SQLite, etc. Semantic Memory allows also to index\n",
    "external data sources, without duplicating all the information, more on that later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "using Microsoft.SemanticKernel.Memory;\n",
    "\n",
    "var (useAzureOpenAI, model, azureEndpoint, apiKey, orgId) = Settings.LoadFromFile();\n",
    "\n",
    "var builder = new KernelBuilder();\n",
    "\n",
    "if (useAzureOpenAI)\n",
    "{\n",
    "    builder.WithAzureTextEmbeddingGenerationService(\"text-embedding-ada-002\", azureEndpoint, apiKey);\n",
    "    builder.WithAzureChatCompletionService(model, azureEndpoint, apiKey);\n",
    "}\n",
    "else\n",
    "{\n",
    "    builder.WithOpenAITextEmbeddingGenerationService(\"text-embedding-ada-002\", apiKey);\n",
    "    builder.WithOpenAIChatCompletionService(model, apiKey, orgId);\n",
    "}\n",
    "\n",
    "builder.WithMemoryStorage(new VolatileMemoryStore());\n",
    "\n",
    "var kernel = builder.Build();"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At its core, Semantic Memory is a set of data structures that allow you to store\n",
    "the meaning of text that come from different data sources, and optionally to store\n",
    "the source text too.\n",
    "\n",
    "These texts can be from the web, e-mail providers, chats, a database, or from your\n",
    "local directory, and are hooked up to the Semantic Kernel through data source connectors.\n",
    "\n",
    "The texts are embedded or compressed into a vector of floats representing mathematically\n",
    "the texts' contents and meaning.\n",
    "\n",
    "You can read more about embeddings [here](https://aka.ms/sk/embeddings)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Manually adding memories\n",
    "Let's create some initial memories \"About Me\". We can add memories to our `VolatileMemoryStore` by using `SaveInformationAsync`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "const string MemoryCollectionName = \"aboutMe\";\n",
    "\n",
    "await kernel.Memory.SaveInformationAsync(MemoryCollectionName, id: \"info1\", text: \"My name is Andrea\");\n",
    "await kernel.Memory.SaveInformationAsync(MemoryCollectionName, id: \"info2\", text: \"I currently work as a tourist operator\");\n",
    "await kernel.Memory.SaveInformationAsync(MemoryCollectionName, id: \"info3\", text: \"I currently live in Seattle and have been living there since 2005\");\n",
    "await kernel.Memory.SaveInformationAsync(MemoryCollectionName, id: \"info4\", text: \"I visited France and Italy five times since 2015\");\n",
    "await kernel.Memory.SaveInformationAsync(MemoryCollectionName, id: \"info5\", text: \"My family is from New York\");"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try searching the memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "var questions = new[]\n",
    "{\n",
    "    \"what is my name?\",\n",
    "    \"where do I live?\",\n",
    "    \"where is my family from?\",\n",
    "    \"where have I travelled?\",\n",
    "    \"what do I do for work?\",\n",
    "};\n",
    "\n",
    "foreach (var q in questions)\n",
    "{\n",
    "    var response = await kernel.Memory.SearchAsync(MemoryCollectionName, q).FirstOrDefaultAsync();\n",
    "    Console.WriteLine(q + \" \" + response?.Metadata.Text);\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now revisit our chat sample from the [previous notebook](04-context-variables-chat.ipynb).\n",
    "If you remember, we used context variables to fill the prompt with a `history` that continuously got populated as we chatted with the bot. Let's add also memory to it!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is done by using the `TextMemorySkill` which exposes the `recall` native function.\n",
    "\n",
    "`recall` takes an input ask and performs a similarity search on the contents that have\n",
    "been embedded in the Memory Store. By default, `recall` returns the most relevant memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "using Microsoft.SemanticKernel.Skills.Core;\n",
    "\n",
    "// TextMemorySkill provides the \"recall\" function\n",
    "kernel.ImportSkill(new TextMemorySkill(kernel.Memory));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "const string skPrompt = @\"\n",
    "ChatBot can have a conversation with you about any topic.\n",
    "It can give explicit instructions or say 'I don't know' if it does not have an answer.\n",
    "\n",
    "Information about me, from previous conversations:\n",
    "- {{$fact1}} {{recall $fact1}}\n",
    "- {{$fact2}} {{recall $fact2}}\n",
    "- {{$fact3}} {{recall $fact3}}\n",
    "- {{$fact4}} {{recall $fact4}}\n",
    "- {{$fact5}} {{recall $fact5}}\n",
    "\n",
    "Chat:\n",
    "{{$history}}\n",
    "User: {{$userInput}}\n",
    "ChatBot: \";\n",
    "\n",
    "var chatFunction = kernel.CreateSemanticFunction(skPrompt, maxTokens: 200, temperature: 0.8);"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `RelevanceParam` is used in memory search and is a measure of the relevance score from 0.0 to 1.0, where 1.0 means a perfect match. We encourage users to experiment with different values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "var context = kernel.CreateNewContext();\n",
    "\n",
    "context[\"fact1\"] = \"what is my name?\";\n",
    "context[\"fact2\"] = \"where do I live?\";\n",
    "context[\"fact3\"] = \"where is my family from?\";\n",
    "context[\"fact4\"] = \"where have I travelled?\";\n",
    "context[\"fact5\"] = \"what do I do for work?\";\n",
    "\n",
    "context[TextMemorySkill.CollectionParam] = MemoryCollectionName;\n",
    "context[TextMemorySkill.RelevanceParam] = \"0.8\";"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've included our memories, let's chat!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "var history = \"\";\n",
    "context[\"history\"] = history;\n",
    "Func<string, Task> Chat = async (string input) => {\n",
    "    // Save new message in the context variables\n",
    "    context[\"userInput\"] = input;\n",
    "\n",
    "    // Process the user message and get an answer\n",
    "    var answer = await chatFunction.InvokeAsync(context);\n",
    "\n",
    "    // Append the new interaction to the chat history\n",
    "    history += $\"\\nUser: {input}\\nChatBot: {answer}\\n\";\n",
    "    context[\"history\"] = history;\n",
    "    \n",
    "    // Show the bot response\n",
    "    Console.WriteLine(\"ChatBot: \" + context);\n",
    "};"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "await Chat(\"Hello, I think we've met before, remember? my name is...\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "await Chat(\"I want to plan a trip and visit my family. Do you know where that is?\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "await Chat(\"Great! What are some fun things to do there?\");"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding documents to your memory\n",
    "\n",
    "Many times in your applications you'll want to bring in external documents into your memory. Let's see how we can do this using our VolatileMemoryStore.\n",
    "\n",
    "Let's first get some data using some of the links in the Semantic Kernel repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "const string memoryCollectionName = \"SKGitHub\";\n",
    "\n",
    "var githubFiles = new Dictionary<string, string>()\n",
    "{\n",
    "    [\"https://github.com/microsoft/semantic-kernel/blob/main/README.md\"]\n",
    "        = \"README: Installation, getting started, and how to contribute\",\n",
    "    [\"https://github.com/microsoft/semantic-kernel/blob/main/dotnet/notebooks/02-running-prompts-from-file.ipynb\"]\n",
    "        = \"Jupyter notebook describing how to pass prompts from a file to a semantic skill or function\",\n",
    "    [\"https://github.com/microsoft/semantic-kernel/blob/main/dotnet/notebooks/00-getting-started.ipynb\"]\n",
    "        = \"Jupyter notebook describing how to get started with the Semantic Kernel\",\n",
    "    [\"https://github.com/microsoft/semantic-kernel/tree/main/samples/skills/ChatSkill/ChatGPT\"]\n",
    "        = \"Sample demonstrating how to create a chat skill interfacing with ChatGPT\",\n",
    "    [\"https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/SemanticKernel/Memory/Volatile/VolatileMemoryStore.cs\"]\n",
    "        = \"C# class that defines a volatile embedding store\",\n",
    "    [\"https://github.com/microsoft/semantic-kernel/tree/main/samples/dotnet/KernelHttpServer/README.md\"]\n",
    "        = \"README: How to set up a Semantic Kernel Service API using Azure Function Runtime v4\",\n",
    "    [\"https://github.com/microsoft/semantic-kernel/tree/main/samples/apps/chat-summary-webapp-react/README.md\"]\n",
    "        = \"README: README associated with a sample starter react-based chat summary webapp\",\n",
    "};"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build a new Kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "var builder = new KernelBuilder();\n",
    "\n",
    "if (useAzureOpenAI)\n",
    "{\n",
    "    builder.WithAzureTextEmbeddingGenerationService(\"text-embedding-ada-002\", azureEndpoint, apiKey);\n",
    "    builder.WithAzureChatCompletionService(model, azureEndpoint, apiKey);\n",
    "}\n",
    "else\n",
    "{\n",
    "    builder.WithOpenAITextEmbeddingGenerationService(\"text-embedding-ada-002\", apiKey);\n",
    "    builder.WithOpenAIChatCompletionService(model, apiKey, orgId);\n",
    "}\n",
    "\n",
    "builder.WithMemoryStorage(new VolatileMemoryStore());\n",
    "\n",
    "var kernel = builder.Build();"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's add these files to our VolatileMemoryStore using `SaveReferenceAsync`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "Console.WriteLine(\"Adding some GitHub file URLs and their descriptions to a volatile Semantic Memory.\");\n",
    "var i = 0;\n",
    "foreach (var entry in githubFiles)\n",
    "{\n",
    "    await kernel.Memory.SaveReferenceAsync(\n",
    "        collection: memoryCollectionName,\n",
    "        description: entry.Value,\n",
    "        text: entry.Value,\n",
    "        externalId: entry.Key,\n",
    "        externalSourceName: \"GitHub\"\n",
    "    );\n",
    "    Console.WriteLine($\"  URL {++i} saved\");\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "string ask = \"I love Jupyter notebooks, how should I get started?\";\n",
    "Console.WriteLine(\"===========================\\n\" +\n",
    "                    \"Query: \" + ask + \"\\n\");\n",
    "\n",
    "var memories = kernel.Memory.SearchAsync(memoryCollectionName, ask, limit: 5, minRelevanceScore: 0.77);\n",
    "\n",
    "i = 0;\n",
    "await foreach (MemoryQueryResult memory in memories)\n",
    "{\n",
    "    Console.WriteLine($\"Result {++i}:\");\n",
    "    Console.WriteLine(\"  URL:     : \" + memory.Metadata.Id);\n",
    "    Console.WriteLine(\"  Title    : \" + memory.Metadata.Description);\n",
    "    Console.WriteLine(\"  Relevance: \" + memory.Relevance);\n",
    "    Console.WriteLine();\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you might be wondering what happens if you have so much data that it doesn't fit into your RAM? That's where you want to make use of an external Vector Database made specifically for storing and retrieving embeddings.\n",
    "\n",
    "Stay tuned for that!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".NET (C#)",
   "language": "C#",
   "name": ".net-csharp"
  },
  "language_info": {
   "name": "polyglot-notebook"
  },
  "polyglot_notebook": {
   "kernelInfo": {
    "defaultKernelName": "csharp",
    "items": [
     {
      "aliases": [],
      "name": "csharp"
     }
    ]
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}