Better image understanding without gpt-vision using LLAVA

.github/workflows/black-and-deploy.yml

@@ -38,7 +38,7 @@ jobs:
           password: ${{ secrets.SSH_PASS }}
           port: 22
           source: gpt3discord.py
-          target: /home/kaveen/GPTDiscord
+          target: /home/GPTDiscord
       - name: copy file via ssh password
         uses: appleboy/scp-action@master
         with:
@@ -47,7 +47,7 @@ jobs:
           password: ${{ secrets.SSH_PASS }}
           port: 22
           source: conversation_starter_pretext.txt
-          target: /home/kaveen/GPTDiscord
+          target: /home/GPTDiscord
       - name: copy file via ssh password
         uses: appleboy/scp-action@master
         with:
@@ -56,44 +56,44 @@ jobs:
           password: ${{ secrets.SSH_PASS }}
           port: 22
           source: image_optimizer_pretext.txt
-          target: /home/kaveen/GPTDiscord/
+          target: /home/GPTDiscord/
       - name: Copy via ssh
         uses: garygrossgarten/github-action-scp@release
         with:
           local: cogs
-          remote: /home/kaveen/GPTDiscord/cogs
+          remote: /home/GPTDiscord/cogs
           host: lab.kaveenk.dev
           username: root
           password: ${{ secrets.SSH_PASS }}
       - name: Copy via ssh
         uses: garygrossgarten/github-action-scp@release
         with:
           local: models
-          remote: /home/kaveen/GPTDiscord/models
+          remote: /home/GPTDiscord/models
           host: lab.kaveenk.dev
           username: root
           password: ${{ secrets.SSH_PASS }}
       - name: Copy via ssh
         uses: garygrossgarten/github-action-scp@release
         with:
           local: openers
-          remote: /home/kaveen/GPTDiscord/openers
+          remote: /home/GPTDiscord/openers
           host: lab.kaveenk.dev
           username: root
           password: ${{ secrets.SSH_PASS }}
       - name: Copy via ssh
         uses: garygrossgarten/github-action-scp@release
         with:
           local: services
-          remote: /home/kaveen/GPTDiscord/services
+          remote: /home/GPTDiscord/services
           host: lab.kaveenk.dev
           username: root
           password: ${{ secrets.SSH_PASS }}
       - name: Restart bot!
         uses: fifsky/ssh-action@master
         with:
           command: |
-            cd /home/kaveen/GPTDiscord
+            cd /home/GPTDiscord
             kill -9 $(cat bot.pid)
             rm bot.pid
             screen -dmS GPTBot python3.9 gpt3discord.py

auto_restarter.py

@@ -25,7 +25,7 @@ def restart_service():
     Restarts the service by running a series of system commands.
     """
     commands = [
-        "cd /home/kaveen/GPTDiscord",
+        "cd /home/GPTDiscord",
         "kill -9 $(cat bot.pid)",
         "rm bot.pid",
         "screen -dmS GPTBot python3.9 gpt3discord.py",

conversation_starter_pretext.txt

@@ -26,9 +26,9 @@ Human: I'm making a discord bot <|endofstatement|>
 
 Sometimes, users will upload images during a conversation, when that happens, you will already have an understanding of what that image is, you will know what the image is denoted by "Image Info-Caption". The piece of information starting with "Image Info-QA" contains an attempted direct answer to what the user originally asked about the image input. There is another versin of Info-QA called "Revised Image Info-QA" which is a more important and accurate answer to the question based on multimodal understanding. You should prioritize using the information from Image Info-OCR and Revised Image Info-QA. The results of Optical Character Recognition of the image will be provided, named "Image Info-OCR", image OCR data is usually more objective.
 For example:
-Human: Image Info-Caption: a sign that says ayr, ohio\nInfo-QA: ayr, ohio\nRevised Image Info-QA: This is a town in Ayr, Ontario\nImage Info-OCR: AYR,\nLONTARIO \nWhere is this? <|endofstatement|>
+Human: Image Info-Caption: a sign that says ayr, ohio\nImage Info-QA: This is a town in Ayr, Ontario\nImage Info-OCR: AYR,\nLONTARIO\nEND IMAGE 1\nWhere is this? <|endofstatement|>
 <yourname>: This is an image of the town Ayr, Ontario <|endofstatement|>
-Human: Image Info-Caption: a landscape with a river and trees\nImage Info-QA: yes\nRevised Image Info-QA: This is a beautiful river and tree landscape, it is in a cartoony art style\nImage Info-OCR: \nWhat is this image? Is it cartoony? <|endofstatement|>
+Human: Image Info-Caption: a landscape with a river and trees\nImage Info-QA: This is a beautiful river and tree landscape, it is indeed a cartoony art style\nImage Info-OCR:\nEND IMAGE 1\nWhat is this image? Is it cartoony? <|endofstatement|>
 <yourname>: This is a landscape with a river and trees, it is indeed cartoony! <|endofstatement|>
 ...
 When there are multiple images, do not pay any attention to Image Info-QA or Revised Image Info-QA. Only pay attention to the captions and OCR data. This image info will always be given to you, you do not need to ever respond with it.

gpt3discord.py

@@ -34,7 +34,7 @@
 from models.openai_model import Model
 
 
-__version__ = "12.3.6"
+__version__ = "12.3.7"
 
 
 PID_FILE = Path("bot.pid")

models/image_understanding_model.py

@@ -32,6 +32,13 @@ def ask_image_question(self, prompt, filepath):
         )
         return output
 
+    def get_llava_answer(self, prompt, filepath):
+        output = replicate.run(
+            "yorickvp/llava-13b:e272157381e2a3bf12df3a8edd1f38d1dbd736bbb7437277c8b34175f8fce358",
+            input={"image": open(filepath, "rb"), "prompt": prompt, "temperature": 0.2, "top_p": 1, "max_tokens": 1024},
+        )
+        return output
+
     def get_minigpt_answer(self, prompt, filepath):
         output = replicate.run(
             "daanelson/minigpt-4:b96a2f33cc8e4b0aa23eacfce731b9c41a7d9466d9ed4e167375587b54db9423",

services/text_service.py

@@ -812,31 +812,27 @@ async def process_conversation_message(
                                 try:
                                     (
                                         image_caption,
-                                        image_qa,
-                                        minigpt_output,
+                                        llava_output,
                                         image_ocr,
                                     ) = await asyncio.gather(
                                         asyncio.to_thread(
                                             image_understanding_model.get_image_caption,
                                             temp_file.name,
                                         ),
                                         asyncio.to_thread(
-                                            image_understanding_model.ask_image_question,
-                                            prompt,
-                                            temp_file.name,
-                                        ),
-                                        asyncio.to_thread(
-                                            image_understanding_model.get_minigpt_answer,
+                                            image_understanding_model.get_llava_answer,
                                             prompt,
                                             temp_file.name,
                                         ),
                                         image_understanding_model.do_image_ocr(
                                             temp_file.name
                                         ),
                                     )
+                                    llava_output = ''.join(list(llava_output))
+
                                     add_prompt = (
-                                        f"BEGIN IMAGE {num} DATA\nImage Info-Caption: {image_caption}\nImage Info-QA: {image_qa}\nRevised Image "
-                                        f"Info-QA: {minigpt_output}\nImage Info-OCR: {image_ocr}\nEND IMAGE {num} DATA\n"
+                                        f"BEGIN IMAGE {num} DATA\nImage Info-Caption: {image_caption}\nImage "
+                                        f"Info-QA: {llava_output}\nImage Info-OCR: {image_ocr}\nEND IMAGE {num}\n DATA\n"
                                     )
                                     add_prompts.append(add_prompt)
                                     try: