diff --git a/README.md b/README.md index 259577f..16a32c3 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ git clone https://github.com/your-username/baby-bliss-bot cd baby-bliss-bot ``` -### Create/Activitate Virtual Environment +### Create/Activate Virtual Environment Always activate and use the python virtual environment to maintain an isolated environment for project's dependencies. * [Create the virtual environment](https://docs.python.org/3/library/venv.html) @@ -58,31 +58,55 @@ with generating new Bliss symbols etc. ### Llama2 -Conclusion: useful +**Conclusion**: useful See the [Llama2FineTuning.md](./docs/Llama2FineTuning.md) in the [documentation](./docs) folder for details on how to fine tune, evaluation results and the conclusion about how useful it is. ### StyleGAN3 -Conclusion: not useful +**Conclusion**: not useful See the [TrainStyleGAN3Model.md](./docs/TrainStyleGAN3Model.md) in the [documentation](./docs) folder for details on how to train this model, training results and the conclusion about how useful it is. ### StyleGAN2-ADA -Conclusion: shows promise +**Conclusion**: shows promise See the [StyleGAN2-ADATraining.md](./docs/StyleGAN2-ADATraining.md) in the [documentation](./docs) folder for details on how to train this model and training results. ### Texture Inversion -Conclusion: not useful +**Conclusion**: not useful See the [Texture Inversion documentation](./notebooks/README.md) for details. +## Preserving Information + +### RAG (Retrieval-augmented generation) + +**Conclusion**: useful + +RAG (Retrieval-augmented generation) technique is explored to resolve ambiguities by retrieving relevant contextual +information from external sources, enabling the language model to generate more accurate and reliable responses. + +See [RAG.md](./docs/RAG.md) for more details. + +### Reflection over Chat History + +**Conclusion**: useful + +When users have a back-and-forth conversation, the application requires a form of "memory" to retain and incorporate past interactions into its current processing. Two methods are explored to achieve this: + +1. Summarizing the chat history and providing it as contextual input. +2. Using prompt engineering to instruct the language model to consider the past conversation. + +The second method, prompt engineering, yields more desired responses than summarizing chat history. + +See [ReflectChatHistory.md](./docs/RAG.md) for more details. + ## Notebooks [`/notebooks`](./notebooks/) directory contains all notebooks used for training or fine-tuning various models. @@ -90,7 +114,8 @@ Each notebook usually comes with a accompanying `dockerfile.yml` to elaborate th running in. ## Jobs -[`/jobs`](./jobs/) directory contains all jobs used for training or fine-tuning various models. +[`/jobs`](./jobs/) directory contains all jobs and scripts used for training or fine-tuning various models, as well +as other explorations with RAG (Retrieval-augmented generation) and preserving chat history. ## Utility Scripts diff --git a/docs/RAG.md b/docs/RAG.md new file mode 100644 index 0000000..8d42b99 --- /dev/null +++ b/docs/RAG.md @@ -0,0 +1,73 @@ +# Experiment with Retrieval-Augumented Generation (RAG) + +Retrieval-augmented generation (RAG) is a technique for enhancing the accuracy and reliability of +generative AI models with facts fetched from external sources. This approach aims to address the +limitations of traditional language models, which may generate responses based solely on their +training data, potentially leading to factual errors or inconsistencies. Read +[What Is Retrieval-Augmented Generation, aka RAG?](https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/) +for more information. + +In a co-design session with an AAC (Augmentative and Alternative Communication) user, RAG can +be particularly useful. When the user expressed a desire to invite "Roy nephew" to her birthday +party, the ambiguity occurred as to whether "Roy" and "nephew" referred to the same person or +different individuals. Traditional language models might interpret this statement inconsistently, +sometimes treating "Roy" and "nephew" as the same person, and other times as separate persons. + +RAG addresses this issue by leveraging external knowledge sources, such as documents or databases +containing relevant information about the user's family members and their relationships. By +retrieving and incorporating this contextual information into the language model's input, RAG +can disambiguate the user's intent and generate a more accurate response. + +The RAG experiment is located in the `jobs/RAG` directory. It contains these scripts: + +* `requirements.txt`: contains python dependencies for setting up the environment to run +the python script. +* `rag.py`: use RAG to address the "Roy nephew" issue described above. + +## Run Scripts Locally + +### Prerequisites + +* If you are currently in a activated virtual environment, deactivate it. + +* Install and start [Ollama](https://github.com/ollama/ollama) to run language models locally + * Follow [README](https://github.com/ollama/ollama?tab=readme-ov-file#customize-a-model) to + install and run Ollama on a local computer. + +* Download a Sentence Transformer Model + 1. Select a Model + - Choose a [sentence transformer model](https://huggingface.co/sentence-transformers) from Hugging Face. + 2. Download the Model + - Make sure that your system has the git-lfs command installed. See + [Git Large File Storage](https://git-lfs.com/) for instructions. + - Download the selected model to a local directory. For example, to download the + [`all-MiniLM-L6-v2` model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2), use the following + command: + ```sh + git clone https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 + ``` + 3. Provide the Model Path + - When running the `rag.py` script, provide the path to the directory of the downloaded model as a parameter. + **Note:** Accessing a local sentence transformer model is much faster than accessing it via the + `sentence-transformers` Python package. + +### Create/Activate Virtual Environment +* Go to the RAG scripts directory + - `cd jobs/RAG` + +* [Create the virtual environment](https://docs.python.org/3/library/venv.html) + (one time setup): + - `python -m venv .venv` + +* Activate (every command-line session): + - Windows: `.\.venv\Scripts\activate` + - Mac/Linux: `source .venv/bin/activate` + +* Install Python Dependencies (Only run once for the installation) + - `pip install -r requirements.txt` + +### Run Scripts +* Run `rag.py` with a parameter providing the path to the directory of a sentence transformer model + - `python rag.py ./all-MiniLM-L6-v2/` + - The last two responses in the execution result shows the language model's output + with and without the use of RAG. diff --git a/docs/ReflectChatHistory.md b/docs/ReflectChatHistory.md new file mode 100644 index 0000000..4046401 --- /dev/null +++ b/docs/ReflectChatHistory.md @@ -0,0 +1,77 @@ +# Reflection over Chat History + +When users have a back-and-forth conversation, the application requires a form of "memory" to retain and incorporate +past interactions into its current processing. Two methods are explored to achieve this: + +1. Summarizing the chat history and providing it as contextual input. +2. Using prompt engineering to instruct the language model to consider the past conversation. + +The second method, prompt engineering, yields more desired responses than summarizing chat history. + +The scripts for this experiment is located in the `jobs/RAG` directory. + +## Method 1: Summarizing the Chat History + +### Steps + +1. Summarize the past conversation and include it in the prompt as contextual information. +2. Include a specified number of the most recent conversation exchanges in the prompt for additional context. +3. Instruct the language model to convert the telegraphic replies from the AAC user into full sentences to continue +the conversation. + +### Result + +The conversion process struggles to effectively utilize the provided summary, often resulting in inaccurate full +sentences. + +### Scripts + +* `requirements.txt`: Lists the Python dependencies needed to set up the environment. +* `chat_history_with_summary.py`: Implements the steps described above and displays the output. + +## Method 2: Using Prompt Engineering + +### Steps + +1. Include the past conversation in the prompt as contextual information. +2. Instruct the language model to reference this context when converting the telegraphic replies from the AAC user +into full sentences to continue the conversation. + +### Result + +The converted sentences are more accurate and appropriate compared to those generated using Method 1. + +### Scripts + +* `requirements.txt`: Lists the Python dependencies needed to set up the environment. +* `chat_history_with_prompt.py`: Implements the steps described above and displays the output. + +## Run Scripts Locally + +### Prerequisites + +* [Ollama](https://github.com/ollama/ollama) to run language models locally + * Follow [README](https://github.com/ollama/ollama?tab=readme-ov-file#customize-a-model) to + install and run Ollama on a local computer. +* If you are currently in a activated virtual environment, deactivate it. + +### Create/Activate Virtual Environment +* Go to the RAG scripts directory + - `cd jobs/RAG` + +* [Create the virtual environment](https://docs.python.org/3/library/venv.html) + (one time setup): + - `python -m venv .venv` + +* Activate (every command-line session): + - Windows: `.\.venv\Scripts\activate` + - Mac/Linux: `source .venv/bin/activate` + +* Install Python Dependencies (Only run once for the installation) + - `pip install -r requirements.txt` + +### Run Scripts +* Run `chat_history_with_summary.py` or `chat_history_with_prompt.py` + - `python chat_history_with_summary.py` or `python chat_history_with_prompt.py` + - The last two responses in the execution result shows the language model's output + with and without the contextual information. diff --git a/jobs/RAG/chat_history_with_prompt.py b/jobs/RAG/chat_history_with_prompt.py new file mode 100644 index 0000000..41004a8 --- /dev/null +++ b/jobs/RAG/chat_history_with_prompt.py @@ -0,0 +1,69 @@ +# Copyright (c) 2024, Inclusive Design Institute +# +# Licensed under the BSD 3-Clause License. You may not use this file except +# in compliance with this License. +# +# You may obtain a copy of the BSD 3-Clause License at +# https://github.com/inclusive-design/baby-bliss-bot/blob/main/LICENSE + +from langchain_community.chat_models import ChatOllama +from langchain_core.output_parsers import StrOutputParser +from langchain_core.prompts import ChatPromptTemplate + +# Define the Ollama model to use +model = "llama3" + +# Telegraphic reply to be translated +message_to_convert = "she love cooking like share recipes" + +# Conversation history +chat_history = [ + "John: Have you heard about the new Italian restaurant downtown?", + "Elaine: Yes, I did! Sarah mentioned it to me yesterday. She said the pasta there is amazing.", + "John: I was thinking of going there this weekend. Want to join?", + "Elaine: That sounds great! Maybe we can invite Sarah too.", + "John: Good idea. By the way, did you catch the latest episode of that mystery series we were discussing last week?", + "Elaine: Oh, the one with the detective in New York? Yes, I watched it last night. It was so intense!", + "John: I know, right? I didn't expect that plot twist at the end. Do you think Sarah has seen it yet?", + "Elaine: I'm not sure. She was pretty busy with work the last time we talked. We should ask her when we see her at the restaurant.", + "John: Definitely. Speaking of Sarah, did she tell you about her trip to Italy next month?", + "Elaine: Yes, she did. She's so excited about it! She's planning to visit a lot of historical sites.", + "John: I bet she'll have a great time. Maybe she can bring back some authentic Italian recipes for us to try.", +] + +# Instantiate the chat model and split the conversation history +llm = ChatOllama(model=model) + +# Create prompt template +prompt_template_with_context = """ +Elaine prefers to talk using telegraphic messages. +Given a chat history and Elaine's latest response which +might reference context in the chat history, convert +Elaine's response to full sentences. Only respond with +converted full sentences. + +Chat history: +{chat_history} + +Elaine's response: +{message_to_convert} +""" + +prompt = ChatPromptTemplate.from_template(prompt_template_with_context) + +# using LangChain Expressive Language (LCEL) chain syntax +chain = prompt | llm | StrOutputParser() + +print("====== Response without chat history ======") + +print(chain.invoke({ + "chat_history": "", + "message_to_convert": message_to_convert +}) + "\n") + +print("====== Response with chat history ======") + +print(chain.invoke({ + "chat_history": "\n".join(chat_history), + "message_to_convert": message_to_convert +}) + "\n") diff --git a/jobs/RAG/chat_history_with_summary.py b/jobs/RAG/chat_history_with_summary.py new file mode 100644 index 0000000..3d1863f --- /dev/null +++ b/jobs/RAG/chat_history_with_summary.py @@ -0,0 +1,100 @@ +# Copyright (c) 2024, Inclusive Design Institute +# +# Licensed under the BSD 3-Clause License. You may not use this file except +# in compliance with this License. +# +# You may obtain a copy of the BSD 3-Clause License at +# https://github.com/inclusive-design/baby-bliss-bot/blob/main/LICENSE + +from langchain_community.chat_models import ChatOllama +from langchain_core.output_parsers import StrOutputParser +from langchain_core.prompts import ChatPromptTemplate + +# Define the Ollama model to use +model = "llama3" + +# Define the number of the most recent chats to be passed in as the most recent chats. +# The summary of chats before the most recent will be passed in as another context element. +num_of_recent_chat = 1 + +# Telegraphic reply to be translated +message_to_convert = "she love cooking like share recipes" + +# Chat history +chat_history = [ + "John: Have you heard about the new Italian restaurant downtown?", + "Elaine: Yes, I did! Sarah mentioned it to me yesterday. She said the pasta there is amazing.", + "John: I was thinking of going there this weekend. Want to join?", + "Elaine: That sounds great! Maybe we can invite Sarah too.", + "John: Good idea. By the way, did you catch the latest episode of that mystery series we were discussing last week?", + "Elaine: Oh, the one with the detective in New York? Yes, I watched it last night. It was so intense!", + "John: I know, right? I didn't expect that plot twist at the end. Do you think Sarah has seen it yet?", + "Elaine: I'm not sure. She was pretty busy with work the last time we talked. We should ask her when we see her at the restaurant.", + "John: Definitely. Speaking of Sarah, did she tell you about her trip to Italy next month?", + "Elaine: Yes, she did. She's so excited about it! She's planning to visit a lot of historical sites.", + "John: I bet she'll have a great time. Maybe she can bring back some authentic Italian recipes for us to try.", +] +recent_chat_array = [] +earlier_chat_array = [] + +# 1. Instantiate the chat model and split the chat history +llm = ChatOllama(model=model) + +if (len(chat_history) > num_of_recent_chat): + recent_chat_array = chat_history[-num_of_recent_chat:] + earlier_chat_array = chat_history[:-num_of_recent_chat] +else: + recent_chat_array = chat_history + earlier_chat_array = [] + +# 2. Summarize earlier chat +if (len(earlier_chat_array) > 0): + summarizer_prompt = ChatPromptTemplate.from_template("Summarize the following chat history. Provide only the summary, without any additional comments or context. \nChat history: {chat_history}") + chain = summarizer_prompt | llm | StrOutputParser() + summary = chain.invoke({ + "chat_history": "\n".join(earlier_chat_array) + }) +print("====== Summary ======") +print(f"{summary}\n") + +# 3. concetenate recent chat into a string +recent_chat_string = "\n".join(recent_chat_array) +print("====== Recent Chat ======") +print(f"{recent_chat_string}\n") + +# Create prompt template +prompt_template_with_context = """ +### Elaine prefers to talk using telegraphic messages. Help to convert Elaine's reply to a chat into full sentences in first-person. Only respond with the converted full sentences. + +### This is the chat summary: + +{summary} + +### This is the most recent chat between Elaine and others: + +{recent_chat} + +### This is Elaine's most recent response to continue the chat. Please convert: +{message_to_convert} +""" + +prompt = ChatPromptTemplate.from_template(prompt_template_with_context) + +# using LangChain Expressive Language (LCEL) chain syntax +chain = prompt | llm | StrOutputParser() + +print("====== Response without chat history ======") + +print(chain.invoke({ + "summary": "", + "recent_chat": recent_chat_string, + "message_to_convert": message_to_convert +}) + "\n") + +print("====== Response with chat history ======") + +print(chain.invoke({ + "summary": summary, + "recent_chat": recent_chat_string, + "message_to_convert": message_to_convert +}) + "\n") diff --git a/jobs/RAG/data/user_doc.txt b/jobs/RAG/data/user_doc.txt new file mode 100644 index 0000000..fdc8338 --- /dev/null +++ b/jobs/RAG/data/user_doc.txt @@ -0,0 +1,39 @@ +My life begins with Bliss. Jane Green introduced Bliss to me. + Mrs. Green was the principle at Virginia Waters School + in St. John’s Newfoundland. She came from England. +Mrs. Green was in the classroom when I entered the room. She taught spelling and math. I was 12 years old when I started at Virginia Waters School. Mrs. Green had two students who couldn’t communicate one of them was me. She had the opportunity to go Toronto and learned about Bliss she came back and showed us about Bliss. She showed us how Bliss worked, I kind of liked and we watch the film about Mr. Bliss. God I must have saw it 100 times. + I grew up with Bliss and I still use it. +Bliss opened many doors. I could communicate my needs through Bliss. + +Jane was someone special to me, she came in my life through out the years. I loved car trips with Jane many of them to McDonalds. + Now those times V.O.C.A. (Voice Output Communication Aids) wasn’t in my world. Jane always took the time to stop and talked. I was out of Mount Pearl by then Jane knew I didn’t like it. +I learn Mrs. Jane Green die + +Explain exon house When all this happened I was living in a institution called Exon House. There nurses and counselors who help with school stuff. I had a close friend named Terry who took me to xhurch and her house. This also when I met my close friend Alen. I remember he had long hair and a piece of rope tied around his wrist.I started using Bliss with Alen. We could talk more than we did before. I remember I had the word ‘pollution’ in my Bliss Book. I didn’t know what that word meant so Alen taught it to me. Alen worked at the Exon House. But before he even came to work there and before I even had heard about Bliss, Alen used to come plan summer day trips for us at Exon House. We would play games with bean bags and go swimming. There was also a memory game, of course I always won. Nobody could beat me. I think Alen is a social worker now. Sometimes I wonder where Aken and the gang are now. + +In 1977 Kathy and Dave took me out of Exon House, which was an institution. They were a couple from Nova Scotia. They were wait on someone when I got home.affter me They thought about getting someone else. They hired a girl named Sue to relieve them one weekend a month and one night a week. Sue was great. She was also from Nova Scotia. I always joked about them being from the Mainland, because I was from Newfoundland Those were good times We had a little puppy named Buddy. Kathy always hit the dog with the newspaper. More girls came to work with us. When sue had leave and then Kathy got pregnant. +After awhile they were thinking about get another girl + +Kathy and I went on a trip to Winnipeg. After I graduated from school, Kathy and I went to Winnipeg. There was a conference. There were many people who used augmentative communication. I met some great people there. There was a sweet little girl, Her name was Louise. I went over to introduce myself to her. She used a wheelchair, She was a very intelligent girl. So We had a big dinner. They were filming us for a documentary. It was on Sue’s wedding. I don’t really know why they were filming her wedding, my guess is that it was because she was a Bliss user and Bliss communication was a huge deal at the time. Sue used her foot to communicate. Sue also worked hard. +Louise’s friend named her “blabber mouth” because she was always talking. Her friend was either her mother or her teacher. + +The building that we stayed in had all kinds of flags. It was white. It had stairs. It was beautiful. There were partitions in the lobby with information about augmentative communication on them for people to look at and find out about on their own. We had barrels of fun. Those lovely times We had dinner I had tto explane what Bliss was to everyone that I met We also took a part in a movie They were a couple with cp. Her name wa sue O’dell. + +Taught bliss to dave and Kathy, even made a bliss cook book with Kathy. I stayed with Kathy and Dave up until when I was 19 and had finished high school school. + +after high school went to OCCC for two weeks in Bloorview. + +The first time I saw Shirley was through Plexiglass. She was pretty. I said, “That couldn’t be Shirley.” Sure enough it was thee one and only Shirley. + +When I was nineteen years old I went to the Ontario Crippled Children’s Centre for one week to be assesed. I thought Shirley was working with me but I got Lynette. +For a few days I saw a social worker and this lady. So Lynette introduced me to herself and I liked her. + +When my week was up I went home. In September I went back for school. Lynnette was my teacher and +I had a teacher named Marnie who taught me math and reading. We had some fun. I really liked her. Lynnette taught me math and how to add and subtract using money. I was staying at the OCCC. I really enjoyed it there. All the girls shared one big room. Every night we ate toast and tea. We had a Halloween party and I won a big teddy bear. +The social worker was Bob Masan, We got to know each other. Bob made me feel comfortable. we laughed. Bob was a sweet person. +A nurse named Vicky. We got along well, When I had a question I always went to her. + + +I learned life skills for three months, cooking, washing clothes. Then I went back home ST.John’s. + +I have a nephew whose name is Roy. I also have a niece. diff --git a/jobs/RAG/rag.py b/jobs/RAG/rag.py new file mode 100644 index 0000000..ba511a0 --- /dev/null +++ b/jobs/RAG/rag.py @@ -0,0 +1,92 @@ +# Copyright (c) 2023-2024, Inclusive Design Institute +# +# Licensed under the BSD 3-Clause License. You may not use this file except +# in compliance with this License. +# +# You may obtain a copy of the BSD 3-Clause License at +# https://github.com/inclusive-design/baby-bliss-bot/blob/main/LICENSE + +import sys +import os +from langchain_community.document_loaders import TextLoader +from langchain_text_splitters import CharacterTextSplitter +from langchain_community.vectorstores import FAISS +from langchain_huggingface import HuggingFaceEmbeddings +from langchain_community.chat_models import ChatOllama +from langchain_core.output_parsers import StrOutputParser +from langchain_core.prompts import ChatPromptTemplate + + +# A utility function that prints the script usage then exit +def printUsageThenExit(): + print("Usage: python rag.py ") + sys.exit() + + +# Read the path to the sentence transformer model +if len(sys.argv) != 2: + printUsageThenExit() +else: + sentence_transformer_dir = sys.argv[1] + if not os.path.isdir(sentence_transformer_dir): + printUsageThenExit() + +# The location of the user document +user_doc = "./data/user_doc.txt" + +loader = TextLoader(user_doc) +documents = loader.load() + +text_splitter = CharacterTextSplitter(chunk_size=200, chunk_overlap=0) +splitted_docs = text_splitter.split_documents(documents) + +# Instantiate the embedding class +embedding_func = HuggingFaceEmbeddings(model_name=sentence_transformer_dir) + +# Load into the vector database +vectordb = FAISS.from_documents(splitted_docs, embedding_func) + +# Create a vector store retriever +retriever = vectordb.as_retriever() + +# query the vector db to test +queries = [ + "Roy nephew", + "high school"] + +for query in queries: + results = retriever.invoke(query) + print(f"====== Test: Similarity search for \"{query}\" ======\n{results[0].page_content}\n\n") + +# Create prompt template +prompt_template_with_context = """ +### [INST] Help to convert Elaine's telegraphic input in the conversation to full sentences in first-person. Only respond with the converted full sentences. Here is context to help: + +{context} + +### Conversation: +{chat} [/INST] + """ + +llm = ChatOllama(model="llama3", system="Elaine is an AAC user who expresses herself telegraphically. She is now in a meeting with Jutta. Below is the conversation in the meeting. Please help to convert what Elaine said to first-person sentences. Only respond with converted sentences.") +prompt = ChatPromptTemplate.from_template(prompt_template_with_context) + +elaine_reply = "Roy nephew" +full_chat = f"Jutta: Elaine, who would you like to invite to your birthday party?\n Elaine: {elaine_reply}." + +# using LangChain Expressive Language (LCEL) chain syntax +chain = prompt | llm | StrOutputParser() + +print("====== Response without RAG ======") + +print(chain.invoke({ + "context": "", + "chat": full_chat +}) + "\n") + +print("====== Response with RAG ======") + +print(chain.invoke({ + "context": retriever.invoke(elaine_reply), + "chat": full_chat +}) + "\n") diff --git a/jobs/RAG/requirements.txt b/jobs/RAG/requirements.txt new file mode 100644 index 0000000..548fa7b --- /dev/null +++ b/jobs/RAG/requirements.txt @@ -0,0 +1,6 @@ +langchain_community +langchain_core +langchain_text_splitters +langchain-huggingface +sentence_transformers +faiss-cpu