mel.chat slows down as chat history grows

[lianggecm]

I'm building a streaming chat interface based on mel.chat. The streaming becomes extremely slow as chat history grows. I did a simple test on the mel.chat. Here's the test code:

@me.page(path="/chat")
def chat():
  mel.chat(transform)

def transform(prompt: str, history: list[mel.ChatMessage]) -> str:
  start_time = time.time()
  yield '\n'+str(start_time)

  for i in range(10):
    yield ' '*100

  end_time = time.time()
  yield '\n'+str(end_time)
  yield '\n\n Duration '+str(end_time - start_time)

Here's a screenshot of the time duration for several chats:

Anyway to mitigate this?

[richard-to]

I tried replicating your example code, but it doesn't seem to be that slow for me.

How slow is the chat getting? Also how many messages does it take for it to noticeably slow down?

[lianggecm]

I used this code to document the issues that I'm experiencing:

import mesop as me
import mesop.labs as mel

import time
@me.page(path="/chat")
def chat():
  mel.chat(transform)

LINES = ' '.join([
  "Mesop is a Python-based UI framework designed to simplify web UI development for engineers without frontend experience.",
  "It leverages the power of the Angular web framework and Angular Material components, allowing rapid construction of web demos and internal tools.",
  "With Mesop, developers can enjoy a fast build-edit-refresh loop thanks to its hot reload feature, making UI tweaks and component integration seamless.",
  "Deployment is straightforward, utilizing standard HTTP technologies.",
  "Mesop's component library aims for comprehensive Angular Material component coverage, enhancing UI flexibility and composability.",
  "It supports custom components for specific use cases, ensuring developers can extend its capabilities to fit their unique requirements.",
  "Mesop's roadmap includes expanding its component library and simplifying the onboarding processs.",
])


def transform(prompt: str, history: list[mel.ChatMessage]) -> str:
  start_time = time.time()
  yield '\n'+'Start time:'+str(start_time)+'\n'

  for i in range(5):
    yield '\n'
    for each in LINES.split(' '):
        time.sleep(0.02)
        yield each + "\n"



  end_time = time.time()
  yield '\n'+'End time:'+str(end_time)
  yield '\n\n Duration '+str(end_time - start_time)

I repeated the same chat interaction 35+ times and recorded the duration for each chat interaction. Here's a chart of the time it takes to complete each chat interaction (x axis is the chat interaction number and y axis is the time duration for that specific chat):

[richard-to]

Thanks for the example (I removed the sleep that you added to make things quicker to test) and chart. That was helpful to see. With the new example, I was able to see what you're saying about the app getting slower and slower as more messages are added.

I tried this with the normal mode and also with the session memory backend. In both cases, each request is definitely getting slower as more messages are added (I used the network tab in Chrome).

It definitely looks like an issue on the server-side.

My current guess is that the diffing algorithms are slow for the use case where we have a list of elements like in chat. Not sure which one:

1. We have a component tree diffing algorithm
2. And we also have diffing on the state object which we added more recently.
3. Potentially could be due to serialization/deserialization (though with memory backend, it doesn't serialize/deserialize), so not as likely.

I'll need to do a closer profiling of the the various stages. I'll also file a bug. Thanks for this.

---

Not ideally, but I think the short term solution is to prune older messages. If that's not acceptable, you could store the messages outside of state (you'd need to keep a create custom chat.py for this). This could solve the problem if the issue is the state diffing algorithm.

[lianggecm]

Thanks. I'll prune older messages.