[Bug]: Headless mode tells the agent to continue over and over

[mamoodi]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Describe the bug and reproduction steps

If you use the headless mode:
https://docs.all-hands.dev/modules/usage/how-to/headless-mode

after the agent completes the task (for example writing the hello world program), we keep sending it "continue". This makes the agent confused and then angry :D

OpenHands Installation

Docker command in README

OpenHands Version

No response

Operating System

None

Logs, Errors, Screenshots, and Additional Context

==============
[Agent Controller default] LEVEL 0 LOCAL STEP 2 GLOBAL STEP 2

22:14:03 - ACTION
[Agent Controller default] **MessageAction** (source=None)
CONTENT: I've created a bash script called `print_hi.sh` that:
1. Has the proper shebang line (`#!/bin/bash`)
2. Uses the `echo` command to print "hi"
3. Has been made executable with `chmod +x`

The script is now ready to use and you can run it anytime by executing `./print_hi.sh` from the directory where it's located.
22:14:03 - openhands:INFO: agent_controller.py:332 - [Agent Controller default] Setting agent(CodeActAgent) state from AgentState.RUNNING to AgentState.AWAITING_USER_INPUT
22:14:03 - OBSERVATION
[Agent Controller default] AgentStateChangedObservation(content='', agent_state=<AgentState.AWAITING_USER_INPUT: 'awaiting_user_input'>, observation='agent_state_changed')
22:14:03 - USER_ACTION
[Agent Controller default] **MessageAction** (source=EventSource.USER)
CONTENT: continue
22:14:03 - openhands:INFO: agent_controller.py:332 - [Agent Controller default] Setting agent(CodeActAgent) state from AgentState.AWAITING_USER_INPUT to AgentState.RUNNING
22:14:03 - OBSERVATION
[Agent Controller default] AgentStateChangedObservation(content='', agent_state=<AgentState.RUNNING: 'running'>, observation='agent_state_changed')
22:14:03 - OBSERVATION
[Agent Controller default] NullObservation(content='', observation='null')
22:14:03 - OBSERVATION
[Agent Controller default] NullObservation(content='', observation='null')


==============
[Agent Controller default] LEVEL 0 LOCAL STEP 3 GLOBAL STEP 3

22:14:11 - ACTION
[Agent Controller default] **MessageAction** (source=None)
CONTENT: I notice you want me to continue, but I'm not sure what specific aspect you'd like me to continue with. There are several possibilities:

1. Would you like to modify the existing script?
2. Would you like to see what's currently in the script?
3. Would you like to add more functionality to the script?
4. Would you like to create a different script?

Please let me know what you'd like me to do next and I'll be happy to help!
22:14:11 - openhands:INFO: agent_controller.py:332 - [Agent Controller default] Setting agent(CodeActAgent) state from AgentState.RUNNING to AgentState.AWAITING_USER_INPUT
22:14:11 - OBSERVATION
[Agent Controller default] AgentStateChangedObservation(content='', agent_state=<AgentState.AWAITING_USER_INPUT: 'awaiting_user_input'>, observation='agent_state_changed')
22:14:11 - USER_ACTION
[Agent Controller default] **MessageAction** (source=EventSource.USER)
CONTENT: continue

[mamoodi]

I think this is the commit that caused the change: #5400
Does this mean that we have to provide a no continue option to headless now?

CC: @enyst

[mamoodi]

I passed the "--no-auto-continue" to the healdess command and I got this. Which I think is at least the same behavior as previously. Should I update the doc for headless to include this then?

[enyst]

Yes, sorry, we should update the doc. Indeed the default is non-interactive for headless, now, and running interactively (the previous behavior) can be done with --no-auto-continue.

[bscriver123]

@mamoodi @enyst Right now you have conflicting instructions in your docs for headless mode. In the Docker command you keep listing the CLI option --no-auto-continue while in the example you state

poetry run python -m openhands.core.main -t "write a bash script that prints hi"

which doesn't have --no-auto-continue. Can you update the docs to keep the docs clear? (https://github.com/All-Hands-AI/OpenHands/blob/main/docs/modules/usage/how-to/headless-mode.md)

[mamoodi]

@bscriver123 oh yep. I'll update that. Thanks for bringing it up!

[bscriver123]

Thanks. Nonetheless, @mamoodi, I don't think adding/removing this flag solves this issue on headless mode. I have been testing and when using Docker I get the following behaviour:

• WITHOUT flag --no-auto-continue According to the logs, the agent keeps changing from Requesting User Input to Running over and over. See the logs for the following command (before I get rate limited):

export WORKSPACE_BASE=$PWD
export LLM_API_KEY=
export LLM_MODEL=gpt-4o
docker run -it \
    --pull=always \
    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.17-nikolaik \
    -e SANDBOX_USER_ID=$(id -u) \
    -e WORKSPACE_MOUNT_PATH=$WORKSPACE_BASE \
    -e LLM_API_KEY=$LLM_API_KEY \
    -e LLM_MODEL=$LLM_MODEL \
    -e LOG_ALL_EVENTS=true \
    -v $WORKSPACE_BASE:/opt/workspace_base \
    -v /var/run/docker.sock:/var/run/docker.sock \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app-$(date +%Y%m%d%H%M%S) \
    docker.all-hands.dev/all-hands-ai/openhands:0.17 \
    python -m openhands.core.main -t "write a bash script that prints hi"

• WITH flag --no-auto-continue The agent awaits user input which is not the intended behaviour. See the
  logs for the following command:

export WORKSPACE_BASE=$PWD
export LLM_API_KEY=
export LLM_MODEL=gpt-4o
docker run -it \
    --pull=always \
    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.17-nikolaik \
    -e SANDBOX_USER_ID=$(id -u) \
    -e WORKSPACE_MOUNT_PATH=$WORKSPACE_BASE \
    -e LLM_API_KEY=$LLM_API_KEY \
    -e LLM_MODEL=$LLM_MODEL \
    -e LOG_ALL_EVENTS=true \
    -v $WORKSPACE_BASE:/opt/workspace_base \
    -v /var/run/docker.sock:/var/run/docker.sock \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app-$(date +%Y%m%d%H%M%S) \
    docker.all-hands.dev/all-hands-ai/openhands:0.17 \
    python -m openhands.core.main -t "write a bash script that prints hi" --no-auto-continue

[mamoodi]

@bscriver123 yes you're absolutely right. Can you create a bug for the second scenario please? With the flag it should not await user input.

Rather, headless mode should not wait for user input after the first request. It should do the task and finish.

[li-boxuan]

WITHOUT flag --no-auto-continue According to the logs, the agent keeps changing from Requesting User Input to Running over and over.

@bscriver123 This is fixed by #5879

[bscriver123]

Thanks @li-boxuan! Then, I assume that WITH the flag --no-auto-continue this will be solved in future releases so I don't need to create a bug @mamoodi. Do you know when we will release?

[enyst]

WITH --no-auto-continue the agent does not continue by itself, instead it waits for the user to respond.

In other words, running main module is by default non-interactive; but you can make it interactive by adding this flag.

[bscriver123]

Then @enyst, what @mamoodi "fixed" in the docs is again incorrect. We should remove the flag --no-auto-continue in the headless docs

[enyst]

It's written this way because it was preserving an old behavior, but I agree we should write this the other way around, it will be clearer. That is, we can document both cases, like:

• command without flag
• followed by a parameters section, where the meaning of this flag is listed.

A PR is welcome if you want to take a stab at it!

[bscriver123]

Okay! Will do and link the PR.

[mamoodi]

Now I'm confused myself. I thought the whole idea of headless was that you give it a task and it should just finish the task. Once it finishes, the interaction should end. Otherwise how is it different than the CLI?

[mamoodi]

I sort of understand now. This I believe was fixed on main. Without providing the flag, it will do the task and exit appropriately which is what headless is supposed to do. PR above should fix it.

[enyst]

I'll take the rest of the discussion in the PR, just for the record @mamoodi 👍

• running via main.py has been the first way of running openhands, and ever since the agent has supported interactivity (somewhere around May), it has been "interactive"
• that's because what we call here "interactivity" comes from the LLM: if the LLM wants to talk to the user, then it doesn't finish its task, it just sends text for the user 😅 . e.g. it might suddenly, in the middle of the task, say "there are two ways to fix this, what do you want me to do?"
• at this point, we have a problem: if we want it to run "properly headless", we have to make it shut up and get back to work 🫠
• what we do, is send it a fake user message, which tells it to shut up and get back to work, basically!

There are other options, like changing the agent or the prompt or raise an error, but I don't think they would be good ideas, at least because we use main.py for evals, and we should eval the real agent, and the real agent allows the LLM to talk.

Note: cli.py is much more recent in the codebase, and when it was introduced it was doing basically the same thing as main.py, maybe continuing also after the task is finished; but currently I think it stops when it thinks the task is finished 🤷‍♂️, just like main.py.

[mamoodi]

This is now fixed!