add_thinking

[Tomlili43]

End-user friendly description of the problem this fixes or functionality that this introduces
Agent thinking feature by thinking

• Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

---

Give a summary of what the PR does, explaining any non-trivial design decisions
add thinking prompt and thinking front end UI.

---

Link of any specific issues this addresses

[neubig]

Hi @Tomlili43 , thanks for the contribution!

In general, all prompts should be implemented in the Python backend, not the frontend. And if we add a new prompt we'll need to validate its effect on agent accuracy and cost.

If you're interested in adding this prompt, could you try adding it in the agenthub/codeact_agent directory? (you can look at the directory structure there to understand what needs to be done)

We would then need to validate the effect of this change on performance and cost. I'm a bit worried that adding this big prompt in the input and encouraging longer outputs would lead to significant increases in time/token cost, but we could try a few examples to see if this makes a big difference in accuracy!

[Tomlili43]

@neubig yeap,

1. added it in the agenthub/codeact_agent directory.
2. for validation, it will cost so many tokens. To be honest , i couldn't afford it. Can you help with it ?

[neubig]

Hey @Tomlili43 , we can try to do this, but it'd be good if you could try running a few examples first to see if it's working!

[Tomlili43]

@neubig hi, validation == run cmd: ./evaluation/agent_bench/scripts/run_infer.sh eval_thinking_prompt_llm HEAD CodeActAgent 1
right ?

[neubig]

Yep, that looks right.

[mamoodi]

@Tomlili43 is this something you still wish to pursue?

[mamoodi]

Going to close this. Please let me know if this was still in progress and we can reopen.