Prompt format for multi-step set up

[Mayer123]

Hi there,

Congratulations on the great work!
I'm curious how should one format the prompt in agent evaluation? i.e. when there are multiple turns of user provided observations and agent actions.
Currently I tried the format below and tested a few tasks on OSWorld, however the results don't look good. The PROMPT_FOR_COMPUTER is just the prompt provided in the readme. So basically I only used the most recent one screenshot and condensed all history actions in the user turn as well.

previous_actions = "\n".join([f"Step {i+1}: {action}" for i, action in enumerate(self.actions)]) if self.actions else "None"
messages = []
messages.append({
    "role": "system",
    "content": [{"type": "text", "text": "You are a helpful assistant."}]
})
messages.append({
    "role": "user",
    "content": [
        {
            "type": "text",
            "text": PROMPT_FOR_COMPUTER + f"{instruction}\nPrevious Actions:\n{previous_actions}" )
        },
        {
            "type": "image_url",
            "image_url": {"url": f"data:image/png;base64,{encode_image(obs['screenshot'])}"}
        }
    ],
})

Could you please share some insights here? Thank you!

[pooruss]

Hi! Here is a pseudocode for the multi step prompt logic:

# To predict third action
messages.append({
    "role": "user",
    "content": [
        {
            "type": "text",
            "text": PROMPT_FOR_COMPUTER + f"{instruction}"
        },
        {
            "type": "image_url",
            "image_url": screenshot_from_init
        },
        {
            "type": "text",
            "text": previous_actions[0],
        },
        {
            "type": "image_url",
            "image_url": screenshot_from_state_0
        },
        {
            "type": "text",
            "text": previous_actions[1],
        },
        {
            "type": "image_url",
            "image_url": screenshot_from_state_1
        }
    ],
})

Note that we apply the 'history 5' logic for multi step online tasks, as discussed in the report.
We will also share our infer codes later. Stay tuned!

[llajan]

Congrats on the great work and thanks for the comments.
When trying the prompt format above, the 72B DPO model complains that "More than 1 image is unsupported". Could you kindly comment on this?

[korbinian-hoermann]

Hi @llajan

Did you try:

python -m vllm.entrypoints.openai.api_server --served-model-name ui-tars --model <path to your model> --limit-mm-per-prompt image=5 -tp <tp>

from:
https://github.com/bytedance/UI-TARS?tab=readme-ov-file#start-an-openai-api-service

[llajan]

That seems to do the job. Thank you!