Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]:Incorrect role in Ollama_model ChatWrapper causing missing assistant messages #441

Closed
cmgzn opened this issue Sep 13, 2024 · 1 comment · Fixed by #443
Closed

[Bug]:Incorrect role in Ollama_model ChatWrapper causing missing assistant messages #441

cmgzn opened this issue Sep 13, 2024 · 1 comment · Fixed by #443
Labels
bug Something isn't working

Comments

@cmgzn
Copy link
Contributor

cmgzn commented Sep 13, 2024

AgentScope is an open-source project. To involve a broader community, we recommend asking your questions in English.

Describe the bug
The docstring for the function OllamaChatWrapper.format states that the role should be "user" in the input messages list. However, the actual implementation sets the role to "system", causing the LLM to only receive system messages and no user messages, and ollama_chat_llama3.1 seems to not respond to system messages in isolation

To Reproduce
Steps to reproduce the behavior:

/examples/conversation_basic/conversation.py

add model_config in agent.scope.init:

agentscope.init(
    model_configs=[
            {
                "model_type": "ollama_chat",
                "config_name": "ollama",
                "model_name": "llama3.1",
            },
    ...
    ]

Start a conversation:

2024-09-13 16:31:17.378 | INFO     | agentscope.models.model:__init__:203 - Initialize model by configuration [ollama]
Assistant: 
User input: Hello
User: Hello
Assistant: 
User input: what's ur name
User: what's ur name
Assistant: 

Debugging found that the message returned by LLM was an empty string.

If OllamaChatWrapper.format is modified according to the documentation string, the content can be output normally.

OllamaChatWrapper.format docstring

The prompt will be as follows:
.. code-block:: python
[
{
"role": "user",
"content": (
"You're a helpful assistant\\n\\n"
"## Conversation History\\n"
"Bob: Hi, how can I help you?\\n"
"user: What's the date today?"
)
}
]

but the actual implementation is:

system_message = {
"role": "system",
"content": system_content,
}
if len(images) != 0:
system_message["images"] = images
return [system_message]

When I use ollama_chat_llama3, this problem does not occur.
Perhaps a more general solution can be adopted, such as:

Fix the role in the implementation to be "user" as described in the docstring.
(Optional) Update the logic to follow a more common pattern where system messages are in the "system" role and conversation history is included in the "user" role. This might improve clarity and consistency.

@zhijianma
Copy link
Collaborator

According to ollama official documentation, there is not much difference between the templates of
llama3 and llama3.1 when there is only one message with the role set to "system".

We successfully tested Llama2, Llama3, Qwen:0.5, and Phi when the role was changed from "system" to "user" in #443 .

Regarding the suggestion to split into system message and user message, we need to conduct more testing before making any modifications.

@zhijianma zhijianma linked a pull request Sep 14, 2024 that will close this issue
4 tasks
@cmgzn cmgzn closed this as completed Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants