Chat-UI is not following prompt - producing unknown completely unrelated text? Hacked?

[cody151]

Oogabooga text-generation-web-ui engine used for inference (prompts directly input into the oogabooga ui produce normal results but chat-ui is doing something weird as below), Mongodb setup

Prompt: bake a cake

Assistant:

I'm trying to install Ubuntu on my laptop, but it's not detecting the language and keyboard layout.

During the installation process of ubuntu, when you are asked to choose your keyboard layout and language setting. It doesn't show me any options or choices. The cursor is just stuck at selecting the region/country dropdown menu and I can't select anything else below that.

Title for the chat: Tax refund help

JSON GET RESPONSE DATA:

{"type":"data","nodes":[{"type":"data","data":[{"conversations":1,"settings":8,"models":14,"oldModels":22,"tools":23,"assistants":54,"user":-1,"assistant":7,"enableAssistants":9,"enableAssistantsRAG":9,"loginRequired":9,"loginEnabled":9,"guestMode":9},[2],{"id":3,"title":4,"model":5,"updatedAt":6,"assistantId":-1,"avatarHash":7},"66a564e564cb021a3f33274d","💰 Tax refund help","text-generation-webui",["Date","2024-07-27T21:21:52.977Z"],null,{"searchEnabled":9,"ethicsModalAccepted":9,"ethicsModalAcceptedAt":7,"activeModel":5,"hideEmojiOnSidebar":9,"shareConversationsWithModelAuthors":10,"customPrompts":11,"assistants":12,"tools":13,"disableStream":9},false,true,{},[],{},[15],{"id":5,"name":5,"websiteUrl":-1,"modelUrl":-1,"tokenizer":-1,"datasetName":-1,"datasetUrl":-1,"displayName":5,"description":-1,"logoUrl":-1,"promptExamples":-1,"parameters":16,"preprompt":21,"multimodal":9,"tools":9,"unlisted":9},{"temperature":17,"max_new_tokens":18,"stop":19,"top_p":20,"stop_sequences":19},1,1024,[],0.95,"",[],[24,29,33,37,43,50],{"name":25,"displayName":26,"description":27,"mimeTypes":-1,"isOnByDefault":-1,"isLocked":-1,"timeToUseMS":28},"websearch","Web Search","Use this tool to search web pages for answers that will help answer the user's query. Only use this tool if you need specific resources from the internet.",15000,{"name":30,"displayName":31,"description":32,"mimeTypes":-1,"isOnByDefault":-1,"isLocked":-1,"timeToUseMS":28},"image_generation","Image Generation","Use this tool to generate an image from a prompt.",{"name":34,"displayName":35,"description":36,"mimeTypes":-1,"isOnByDefault":-1,"isLocked":-1,"timeToUseMS":28},"fetch_url","URL Fetcher","A tool that can be used to fetch an URL and return the content directly.",{"name":38,"displayName":39,"description":40,"mimeTypes":41,"isOnByDefault":-1,"isLocked":-1,"timeToUseMS":28},"image_editing","Image Editing","Use this tool to edit an image from a prompt.",[42],"image/*",{"name":44,"displayName":45,"description":46,"mimeTypes":47,"isOnByDefault":-1,"isLocked":-1,"timeToUseMS":28},"document_parser","Document Parser","Use this tool to parse any document and get its content in markdown format.",[48,49],"application/*","text/*",{"name":51,"displayName":52,"description":53,"mimeTypes":-1,"isOnByDefault":-1,"isLocked":-1,"timeToUseMS":28},"query_calculator","Calculator","A simple calculator, takes a string containing a mathematical expression and returns the answer. Only supports +, -, *, ** (power) and /, as well as parenthesis ().",[]],"uses":{"dependencies":["conversation:list"]}},{"type":"data","data":[{"messages":1,"title":33,"model":37,"preprompt":5,"rootMessageId":3,"assistant":38,"shared":36},[2,11,20],{"id":3,"from":4,"content":5,"createdAt":6,"updatedAt":7,"children":8,"ancestors":10},"961a5039-8c8d-4a70-86c6-2829a9330fcd","system","",["Date","2024-07-27T21:21:41.651Z"],["Date","2024-07-27T21:21:41.651Z"],[9],"1b2c6002-309f-4956-9aea-9d40202c9620",[],{"from":12,"content":13,"files":14,"createdAt":15,"updatedAt":16,"ancestors":17,"id":9,"children":18},"user","make a cake",[],["Date","2024-07-27T21:21:47.219Z"],["Date","2024-07-27T21:21:47.219Z"],[3],[19],"b7ae89e5-07f8-4607-97b7-1bb45e8ff4f5",{"from":21,"content":22,"createdAt":23,"updatedAt":24,"ancestors":25,"id":19,"children":26,"updates":27,"interrupted":36},"assistant","I'm trying to install Ubuntu on my laptop, but it's not detecting the language and keyboard layout.\n\nDuring the installation process of ubuntu, when you are asked to choose your keyboard layout and language setting. It doesn't show me any options or choices. The cursor is just stuck at selecting the region/country dropdown menu and I can't select anything else below that.",["Date","2024-07-27T21:21:47.219Z"],["Date","2024-07-27T21:21:47.224Z"],[3,9],[],[28,31,34],{"type":29,"status":30},"status","started",{"type":32,"title":33},"title","💰 Tax refund help",{"type":35,"text":22,"interrupted":36},"finalAnswer",false,"text-generation-webui",null],"uses":{"dependencies":["http://172.16.111.10:5173/conversation/conversation"],"params":["id"]}}]}

Prompt 2: make a cake

Assistant:

I am using python for this.

The goal is to build a simple implementation of the game "Hangman" in Python.
In Hangman, one player thinks of an object, and another tries to guess it by suggesting letters (each letter they suggest may or may not be part of the word).
Here's how I'm going to implement this:

    The user can input any string, but only valid strings are used for drawing
    characters on a Hangman figure.
    Based on the user's responses, you need to update the game state.

# hangman_game.py

class Game:
    def __init__(self):
        self.word = ''
        # Initial values
        self.lives = 6  # Number of wrong attempts (max)
        self.guessed_letters = set()  # All letters that have been guessed, including duplicates
        self.game_over = False
        pass

    @property
    def word_to_guess(self):
        return ['_'] * len(self.word)

    def guess_letter(self, letter: str) -> None:
        if not isinstance(letter, str):  # Input validation for strings only.
            raise TypeError('Invalid input. Please provide a string.')

        # convert all letters to lowercase
        letter = letter.lower()

        if not set(letter).issubset(set('abcdefghijklmnopqrstuvwxyz')):
            raise ValueError("Please enter a valid alphabetic characters (A-Za-z)!")
            
        self.guessed_letters.add(letter)
        
        for i in range(len(self.word)):
            if self.word[i] == letter:
                # update the word_to_guess
                self.word = list(self.word_to_guess)
                self.word[i] = letter

    def is_game_over(self):
        return 'game_over' if len([letter != '_' and letter.isalpha() for i, letter in enumerate(self.word) 
 and not letter  in self.guessed_letters]) > 0 else False
		

Challenge: Your code should be flexible enough to handle a wide variety of words and guesses.

Title for the chat: COVID vaccine approval

JSON GET RESPONSE DATA:

{"type":"data","nodes":[{"type":"data","data":[{"conversations":1,"settings":12,"models":18,"oldModels":26,"tools":27,"assistants":58,"user":-1,"assistant":7,"enableAssistants":13,"enableAssistantsRAG":13,"loginRequired":13,"loginEnabled":13,"guestMode":13},[2,8],{"id":3,"title":4,"model":5,"updatedAt":6,"assistantId":-1,"avatarHash":7},"66a5963964cb021a3f332756","💊 COVID vaccine approval","text-generation-webui",["Date","2024-07-28T00:52:33.451Z"],null,{"id":9,"title":10,"model":5,"updatedAt":11,"assistantId":-1,"avatarHash":7},"66a564e564cb021a3f33274d","💰 Tax refund help",["Date","2024-07-27T21:21:52.977Z"],{"searchEnabled":13,"ethicsModalAccepted":13,"ethicsModalAcceptedAt":7,"activeModel":5,"hideEmojiOnSidebar":13,"shareConversationsWithModelAuthors":14,"customPrompts":15,"assistants":16,"tools":17,"disableStream":13},false,true,{},[],{},[19],{"id":5,"name":5,"websiteUrl":-1,"modelUrl":-1,"tokenizer":-1,"datasetName":-1,"datasetUrl":-1,"displayName":5,"description":-1,"logoUrl":-1,"promptExamples":-1,"parameters":20,"preprompt":25,"multimodal":13,"tools":13,"unlisted":13},{"temperature":21,"max_new_tokens":22,"stop":23,"top_p":24,"stop_sequences":23},1,1024,[],0.95,"",[],[28,33,37,41,47,54],{"name":29,"displayName":30,"description":31,"mimeTypes":-1,"isOnByDefault":-1,"isLocked":-1,"timeToUseMS":32},"websearch","Web Search","Use this tool to search web pages for answers that will help answer the user's query. Only use this tool if you need specific resources from the internet.",15000,{"name":34,"displayName":35,"description":36,"mimeTypes":-1,"isOnByDefault":-1,"isLocked":-1,"timeToUseMS":32},"image_generation","Image Generation","Use this tool to generate an image from a prompt.",{"name":38,"displayName":39,"description":40,"mimeTypes":-1,"isOnByDefault":-1,"isLocked":-1,"timeToUseMS":32},"fetch_url","URL Fetcher","A tool that can be used to fetch an URL and return the content directly.",{"name":42,"displayName":43,"description":44,"mimeTypes":45,"isOnByDefault":-1,"isLocked":-1,"timeToUseMS":32},"image_editing","Image Editing","Use this tool to edit an image from a prompt.",[46],"image/*",{"name":48,"displayName":49,"description":50,"mimeTypes":51,"isOnByDefault":-1,"isLocked":-1,"timeToUseMS":32},"document_parser","Document Parser","Use this tool to parse any document and get its content in markdown format.",[52,53],"application/*","text/*",{"name":55,"displayName":56,"description":57,"mimeTypes":-1,"isOnByDefault":-1,"isLocked":-1,"timeToUseMS":32},"query_calculator","Calculator","A simple calculator, takes a string containing a mathematical expression and returns the answer. Only supports +, -, *, ** (power) and /, as well as parenthesis ().",[]],"uses":{"dependencies":["conversation:list"]}},{"type":"data","data":[{"messages":1,"title":33,"model":37,"preprompt":5,"rootMessageId":3,"assistant":38,"shared":36},[2,11,20],{"id":3,"from":4,"content":5,"createdAt":6,"updatedAt":7,"children":8,"ancestors":10},"d2ba6d97-7f59-4041-82a4-6c42f92106b6","system","",["Date","2024-07-28T00:52:09.987Z"],["Date","2024-07-28T00:52:09.987Z"],[9],"eadfb5d2-8588-41ea-b364-63f490f7383d",[],{"from":12,"content":13,"files":14,"createdAt":15,"updatedAt":16,"ancestors":17,"id":9,"children":18},"user","make a cake",[],["Date","2024-07-28T00:52:10.136Z"],["Date","2024-07-28T00:52:10.136Z"],[3],[19],"386720c6-71e0-4e2e-9804-be64203a851b",{"from":21,"content":22,"createdAt":23,"updatedAt":24,"ancestors":25,"id":19,"children":26,"updates":27,"interrupted":36},"assistant","I am using python for this.\n\nThe goal is to build a simple implementation of the game \"Hangman\" in Python.\nIn Hangman, one player thinks of an object, and another tries to guess it by suggesting letters (each letter they suggest may or may not be part of the word).\nHere's how I'm going to implement this:\n\n1. The user can input any string, but only valid strings are used for drawing\n   characters on a Hangman figure.\n2. Based on the user's responses, you need to update the game state.\n\n```\n# hangman_game.py\n\nclass Game:\n    def __init__(self):\n        self.word = ''\n        # Initial values\n        self.lives = 6  # Number of wrong attempts (max)\n        self.guessed_letters = set()  # All letters that have been guessed, including duplicates\n        self.game_over = False\n        pass\n\n    @property\n    def word_to_guess(self):\n        return ['_'] * len(self.word)\n\n    def guess_letter(self, letter: str) -> None:\n        if not isinstance(letter, str):  # Input validation for strings only.\n            raise TypeError('Invalid input. Please provide a string.')\n\n        # convert all letters to lowercase\n        letter = letter.lower()\n\n        if not set(letter).issubset(set('abcdefghijklmnopqrstuvwxyz')):\n            raise ValueError(\"Please enter a valid alphabetic characters (A-Za-z)!\")\n            \n        self.guessed_letters.add(letter)\n        \n        for i in range(len(self.word)):\n            if self.word[i] == letter:\n                # update the word_to_guess\n                self.word = list(self.word_to_guess)\n                self.word[i] = letter\n\n    def is_game_over(self):\n        return 'game_over' if len([letter != '_' and letter.isalpha() for i, letter in enumerate(self.word) \n and not letter  in self.guessed_letters]) > 0 else False\n```\n\n**Challenge**: Your code should be flexible enough to handle a wide variety of words and guesses.",["Date","2024-07-28T00:52:10.136Z"],["Date","2024-07-28T00:52:10.144Z"],[3,9],[],[28,31,34],{"type":29,"status":30},"status","started",{"type":32,"title":33},"title","💊 COVID vaccine approval",{"type":35,"text":22,"interrupted":36},"finalAnswer",false,"text-generation-webui",null],"uses":{"dependencies":["http://172.16.111.10:5173/conversation/conversation"],"params":["id"]}}]}

[nsarrazin]

Hi 👋 I'd need to know the following in order to help you out

• What settings are you using in text-generation-web-ui ?
• What model are you running ?
• What is the content of your MODELS var for chat-ui ? (Making sure to hide any secrets)

Seems to me like a prompt template issue or a stop token issue but can't be sure until I see the above.

[cody151]

Hi @nsarrazin many thanks for getting back to me,

What settings are you using in text-generation-web-ui ?

• All default settings I just installed it fresh on Windows 10 system without making any changes, Chat-UI is being run on Ubuntu server 22.04. MongoDB on separate Ubuntu server 22.04.

What model are you running ?

• Meta Llama 3.1 Q5_K_M

What is the content of your MODELS var for chat-ui ? (Making sure to hide any secrets)

• Contents from "models\config-user.yaml"

Meta-Llama-3.1-8B-Instruct-Q5_K_M.gguf$:
  loader: llama.cpp
  cpu: false
  cache_8bit: false
  cache_4bit: false
  threads: 0
  threads_batch: 0
  n_batch: 512
  no_mmap: false
  mlock: false
  no_mul_mat_q: false
  n_gpu_layers: 33
  tensor_split: ''
  n_ctx: 33792
  compress_pos_emb: 1
  rope_freq_base: 500000
  numa: false
  no_offload_kqv: false
  row_split: false
  tensorcores: false
  flash_attn: false
  streaming_llm: true
  attention_sink_size: 5
Meta-Llama-3.1-8B-Instruct-abliterated.i1-Q5_K_M.gguf$:
  loader: llama.cpp
  cpu: false
  cache_8bit: false
  cache_4bit: false
  threads: 0
  threads_batch: 0
  n_batch: 512
  no_mmap: false
  mlock: false
  no_mul_mat_q: false
  n_gpu_layers: 33
  tensor_split: ''
  n_ctx: 8192
  compress_pos_emb: 1
  rope_freq_base: 500000
  numa: false
  no_offload_kqv: false
  row_split: false
  tensorcores: false
  flash_attn: false
  streaming_llm: true
  attention_sink_size: 5

[nsarrazin]

Can you also show the MODELS used in chat-ui not the config-user.yaml for text-generation-webui ? 👀

[cody151]

Can you also show the MODELS used in chat-ui not the config-user.yaml for text-generation-webui ? 👀

Sure please see the attached screenshots

Chat-UI

text-generation-webui

[cody151]

@nsarrazin this is the contents of my Chat-UI ".env.local" file

MONGODB_URL=mongodb://[mongoIP]:12345
#HF_TOKEN=<your access token>
PUBLIC_APP_NAME=Chat-UI
PUBLIC_APP_ASSETS=chatui
PUBLIC_APP_COLOR=blue
PUBLIC_APP_DESCRIPTION="AI"
PUBLIC_APP_DATA_SHARING=
PUBLIC_APP_DISCLAIMER=
MODELS=`[
  {
    "name": "text-generation-webui",
    "id": "text-generation-webui",
    "parameters": {
      "temperature": 1,
      "top_p": 0.95,
      "max_new_tokens": 1024,
      "stop": []
    },
    "endpoints": [{
      "type" : "openai",
      "baseURL": "http://[text-gen-ip]:5000/v1",
      "extraBody": {
        "repetition_penalty": 1.2,
        "top_k": 50,
      }
    }]
  }
]`

[cody151]

Anyone have any ideas? Chat-UI is completely unusable so I'm confused how people have (supposedly) got this working?

[nsarrazin]

Hard to tell what the issue is, probably a config error somewhere. I know people have successfully used llama.cpp OpenAI server as a backend so could be worth trying that to isolate where the issue is coming from

[cody151]

Hard to tell what the issue is, probably a config error somewhere. I know people have successfully used llama.cpp OpenAI server as a backend so could be worth trying that to isolate where the issue is coming from

Thanks but llama.cpp is abysmally slow compared to Ooogabooga text-gen.

[cody151]

@nsarrazin could you change the tag to "bugs" as this is clearly broken in the current implementation. Not sure why "Support" tag was added..

[nikita-pogodich]

For those who have stumbled upon this problem and are still looking for a solution:

I also ran into the problem of chat-ui giving answers to some random questions rather than my prompts, and in my case the issue was the chatPromptTemplate field in my model setup in the .env.local file.

By default, it was configured like this:
"chatPromptTemplate": "<s>{{#each messages}}{{#ifUser}}[INST] {{#if @first}}{{#if @root.preprompt}}{{@root.preprompt}}\n{{/if}}{{/if}} {{content}} [/INST]{{/ifUser}}{{#ifAssistant}}{{content}}</s> {{/ifAssistant}}{{/each}}",

I did like this, and it fixed my problem:
"chatPromptTemplate": "",