Unify output handlers and regular tools, improve exception management…

…, major refactor (#76)

* Unify output handlers and regular tools, improve exception management, major refactor

* update tests data

* Add AuxPrompts class for handling auxiliary error messages in MotleyCrew.

* Refactor output handling and documentation in MotleyCrew, enhancing support for direct returns and exception reflection in tools and agents.

* Update tests data

* Update quickstart docs

* bump version

examples/Quickstart.ipynb

@@ -30,16 +30,45 @@
     "Motleycrew provides thin wrappers for all the common agent frameworks: Langchain, LlamaIndex, CrewAI, and Autogen (please let us know if you want any others added!).\n",
     "It also provides thin wrappers for Langchain and LlamaIndex tools, allowing you to use any of these tools with any of these agents.\n",
     "\n",
-    "Motleycrew also supports **delegation**: you can simply give any agent as a tool to any other agent. \n",
+    "MotleyCrew also supports **delegation**: you can simply give any agent as a tool to any other agent. \n",
     "\n",
     "All the wrappers for tools and agents implement the Runnable interface, so you can use them as-is in LCEL and Langgraph code.\n",
     "\n",
     "\n",
-    "### Output handlers\n",
+    "### Output handlers (aka return_direct)\n",
     "\n",
-    "An **output handler** is a special tool that the agent uses for submitting the final output instead of returning it raw. Besides defining a schema for the output, the output handler enables you to implement any validation logic inside it, including agent-based. If your agent returns invalid output, you can raise an exception that will be returned to the agent so that it can retry.\n",
+    "An **output handler** is a tool that the agent uses for submitting the final output instead of returning it raw. Besides defining a schema for the output, the output handler enables you to implement any validation logic inside it, including agent-based. If your agent returns invalid output, you can raise an exception that will be returned to the agent so that it can retry.\n",
     "\n",
-    "See our usage examples with a [simple validator](examples/validating_agent_output.html) and an [advanced output handler with multiple fields](examples/advanced_output_handling.html).\n"
+    "Essentially, an output handler is a tool that returns its output directly to the user, thus finishing the agent execution. This behavior is enabled by setting the `return_direct=True` argument for the tool. Unlike other frameworks, MotleyCrew allows to have multiple output handlers for one agent, from which the agent can choose one.\n",
+    "\n",
+    "MotleyCrew also supports **forced output handlers**. This means that the agent will only be able to return output via an output handler, and not directly. This is useful if you want to ensure that the agent only returns output in a specific format.\n",
+    "\n",
+    "See our usage examples with a [simple validator](examples/validating_agent_output.html) and an [advanced output handler with multiple fields](examples/advanced_output_handling.html).\n",
+    "\n",
+    "\n",
+    "### MotleyTool\n",
+    "\n",
+    "A tool in motleycrew, like in other frameworks, is basically a function that takes an input and returns an output.\n",
+    "It is called a tool in the sense that it is usually used by an agent to perform a specific action.\n",
+    "Besides the function itself, a tool also contains an input schema which describes the input format to the LLM.\n",
+    "\n",
+    "`MotleyTool` is the base class for all tools in motleycrew. It is a subclass of `Runnable` that adds some additional features to the tool, along with necessary adapters and converters.\n",
+    "\n",
+    "If you pass a tool from a supported framework (currently Langchain, LlamaIndex, and CrewAI) to a motleycrew agent, it will be automatically converted. If you want to have control over this, e.g. to customize tool params, you can do it manually.\n",
+    "\n",
+    "```python\n",
+    "motley_tool = MotleyTool.from_supported_tool(my_tool)\n",
+    "```\n",
+    "\n",
+    "It is also possible to define a custom tool using the `MotleyTool` base class, overriding the `run` method. This is especially useful if you want to access context such as the caller agent or its last input, which can be useful for validation.\n",
+    "\n",
+    "```python\n",
+    "class MyTool(MotleyTool):\n",
+    "    def run(self, some_input: str) -> str:\n",
+    "        return f\"Received {some_input} from agent {self.agent} with last input {self.agent_input}\"\n",
+    "```\n",
+    "\n",
+    "MotleyTool can reflect exceptions that are raised inside it back to the agent, which can then retry the tool call. You can pass a list of exception classes to the `exceptions_to_reflect` argument in the constructor (or even pass the `Exception` class to reflect everything)."
    ]
   },
   {

examples/llama_index_output_handler.py

@@ -8,33 +8,29 @@
 from motleycrew.tasks import SimpleTask
 from motleycrew.common.exceptions import InvalidOutput
 from motleycrew.common import AsyncBackend
-
-from langchain_core.tools import StructuredTool
+from motleycrew.tools import MotleyTool
 
 
 def main():
     """Main function of running the example."""
     search_tool = DuckDuckGoSearchRun()
 
-    def check_output(output: str):
-        if "medicine" not in output.lower():
-            raise InvalidOutput(
-                "Add more information about AI applications in medicine."
-            )
+    class OutputHandler(MotleyTool):
+        def run(self, output: str):
+            if "medicine" not in output.lower():
+                raise InvalidOutput("Add more information about AI applications in medicine.")
 
-        return {"checked_output": output}
+            return {"checked_output": output}
 
-    output_handler = StructuredTool.from_function(
-        name="output_handler",
-        description="Output handler",
-        func=check_output,
+    output_handler = OutputHandler(
+        name="output_handler", description="Output handler", return_direct=True
     )
 
     # TODO: add LlamaIndex native tools
     researcher = ReActLlamaIndexMotleyAgent(
         prompt_prefix="Your goal is to uncover cutting-edge developments in AI and data science",
-        tools=[search_tool],
-        output_handler=output_handler,
+        tools=[search_tool, output_handler],
+        force_output_handler=True,
         verbose=True,
         max_iterations=16,  # default is 10, we add more because the output handler may reject the output
     )

motleycrew/agents/__init__.py

@@ -1,13 +1,11 @@
 """Everything agent-related: wrappers, pre-made agents, output handlers etc."""
 
 from .abstract_parent import MotleyAgentAbstractParent
-from .output_handler import MotleyOutputHandler
 from .parent import MotleyAgentParent
 from .langchain import LangchainMotleyAgent
 
 __all__ = [
     "MotleyAgentAbstractParent",
     "MotleyAgentParent",
-    "MotleyOutputHandler",
     "LangchainMotleyAgent",
 ]

motleycrew/agents/abstract_parent.py

@@ -23,10 +23,6 @@ def invoke(
         pass
 
     @abstractmethod
-    def as_tool(self) -> "MotleyTool":
-        """Convert the agent to a MotleyTool to be used by other agents via delegation.
-
-        Returns:
-            The tool representation of the agent.
-        """
+    def call_as_tool(self, *args, **kwargs) -> Any:
+        """Method that is called when the agent is used as a tool by another agent."""
         pass

motleycrew/agents/crewai/crewai.py

@@ -31,7 +31,7 @@ def __init__(
         name: str | None = None,
         agent_factory: MotleyAgentFactory[CrewAIAgentWithConfig] | None = None,
         tools: Sequence[MotleySupportedTool] | None = None,
-        output_handler: MotleySupportedTool | None = None,
+        force_output_handler: bool = False,
         verbose: bool = False,
     ):
         """
@@ -63,14 +63,15 @@ def __init__(
 
             tools: Tools to add to the agent.
 
-            output_handler: Output handler for the agent.
+            force_output_handler: Whether to force the agent to return through an output handler.
+                NOTE: This is currently not supported for CrewAI agents.
 
             verbose: Whether to log verbose output.
         """
 
-        if output_handler:
+        if force_output_handler:
             raise NotImplementedError(
-                "Output handler is not supported for CrewAI agents "
+                "Forced output handlers are not supported for CrewAI agents "
                 "because of the specificity of CrewAI's prompts."
             )
 
@@ -81,7 +82,7 @@ def __init__(
             name=name,
             agent_factory=agent_factory,
             tools=tools,
-            output_handler=output_handler,
+            force_output_handler=False,
             verbose=verbose,
         )
 

motleycrew/agents/crewai/crewai_agent.py

@@ -26,8 +26,8 @@ def __init__(
         description: str | None = None,
         delegation: bool = False,
         tools: Sequence[MotleySupportedTool] | None = None,
+        force_output_handler: bool = False,
         llm: Optional[Any] = None,
-        output_handler: MotleySupportedTool | None = None,
         verbose: bool = False,
     ):
         """
@@ -55,7 +55,8 @@ def __init__(
 
             llm: LLM instance to use.
 
-            output_handler: Output handler for the agent.
+            force_output_handler: Whether to force the use of an output handler.
+                NOTE: This is currently not supported for CrewAI agents.
 
             verbose: Whether to log verbose output.
         """
@@ -91,6 +92,6 @@ def agent_factory(tools: dict[str, MotleyTool]):
             name=role,
             agent_factory=agent_factory,
             tools=tools,
-            output_handler=output_handler,
+            force_output_handler=force_output_handler,
             verbose=verbose,
         )

motleycrew/agents/langchain/langchain.py

@@ -26,7 +26,7 @@ def __init__(
         prompt_prefix: str | ChatPromptTemplate | None = None,
         agent_factory: MotleyAgentFactory[AgentExecutor] | None = None,
         tools: Sequence[MotleySupportedTool] | None = None,
-        output_handler: MotleySupportedTool | None = None,
+        force_output_handler: bool = False,
         chat_history: bool | GetSessionHistoryCallable = True,
         input_as_messages: bool = False,
         runnable_config: RunnableConfig | None = None,
@@ -60,7 +60,8 @@ def __init__(
 
             tools: Tools to add to the agent.
 
-            output_handler: Output handler for the agent.
+            force_output_handler: Whether to force the agent to return through an output handler.
+                If True, at least one tool must have return_direct set to True.
 
             chat_history: Whether to use chat history or not.
                 If `True`, uses `InMemoryChatMessageHistory`.
@@ -82,12 +83,10 @@ def __init__(
             name=name,
             agent_factory=agent_factory,
             tools=tools,
-            output_handler=output_handler,
+            force_output_handler=force_output_handler,
             verbose=verbose,
         )
 
-        self._agent_finish_blocker_tool = self._create_agent_finish_blocker_tool()
-
         if chat_history is True:
             chat_history = InMemoryChatMessageHistory()
             self.get_session_history_callable = lambda _: chat_history
@@ -97,6 +96,8 @@ def __init__(
         self.input_as_messages = input_as_messages
         self.runnable_config = runnable_config
 
+        self._create_agent_error_tool()
+
     def materialize(self):
         """Materialize the agent and wrap it in RunnableWithMessageHistory if needed."""
         if self.is_materialized:
@@ -105,8 +106,9 @@ def materialize(self):
         super().materialize()
         assert isinstance(self._agent, AgentExecutor)
 
-        if self.output_handler:
-            self._agent.tools += [self._agent_finish_blocker_tool]
+        if self.get_output_handlers():
+            assert self._agent_error_tool
+            self._agent.tools += [self._agent_error_tool]
 
             object.__setattr__(
                 self._agent.agent, "plan", self.agent_plan_decorator(self._agent.agent.plan)
@@ -118,22 +120,19 @@ def materialize(self):
                 self.take_next_step_decorator(self._agent._take_next_step),
             )
 
-            prepared_output_handler = None
             for tool in self.agent.tools:
-                if tool.name == self.output_handler.name:
-                    prepared_output_handler = tool
-
-            object.__setattr__(
-                prepared_output_handler,
-                "_run",
-                self._run_tool_direct_decorator(prepared_output_handler._run),
-            )
-
-            object.__setattr__(
-                prepared_output_handler,
-                "run",
-                self.run_tool_direct_decorator(prepared_output_handler.run),
-            )
+                if tool.return_direct:
+                    object.__setattr__(
+                        tool,
+                        "_run",
+                        self._run_tool_direct_decorator(tool._run),
+                    )
+
+                    object.__setattr__(
+                        tool,
+                        "run",
+                        self.run_tool_direct_decorator(tool.run),
+                    )
 
         if self.get_session_history_callable:
             logger.info("Wrapping agent in RunnableWithMessageHistory")

motleycrew/agents/langchain/legacy_react.py

@@ -14,10 +14,10 @@
 from motleycrew.common.llms import init_llm
 from motleycrew.tools import MotleyTool
 
-OUTPUT_HANDLER_WITH_DEFAULT_PROMPT_MESSAGE = (
+FORCED_OUTPUT_HANDLER_WITH_DEFAULT_PROMPT_MESSAGE = (
     "Langchain's default ReAct prompt tells the agent to include a final answer keyword, "
-    "which later confuses the parser when an output handler is used. "
-    "Please provide a custom prompt if using an output handler."
+    "which later confuses the agent when an output handler is used. "
+    "Please provide a custom prompt if forcing an output handler."
 )
 
 
@@ -34,8 +34,8 @@ def __init__(
         description: str | None = None,
         name: str | None = None,
         prompt_prefix: str | None = None,
-        output_handler: MotleySupportedTool | None = None,
         chat_history: bool | GetSessionHistoryCallable = True,
+        force_output_handler: bool = False,
         prompt: str | None = None,
         handle_parsing_errors: bool = True,
         handle_tool_errors: bool = True,
@@ -51,6 +51,7 @@ def __init__(
             prompt_prefix: Prefix to the agent's prompt.
             output_handler: Output handler for the agent.
             chat_history: Whether to use chat history or not.
+            force_output_handler: Whether to force the agent to return through an output handler.
             prompt: Custom prompt to use with the agent.
             handle_parsing_errors: Whether to handle parsing errors.
             handle_tool_errors: Whether to handle tool errors.
@@ -60,8 +61,8 @@ def __init__(
             verbose: Whether to log verbose output.
         """
         if prompt is None:
-            if output_handler is not None:
-                raise Exception(OUTPUT_HANDLER_WITH_DEFAULT_PROMPT_MESSAGE)
+            if force_output_handler:
+                raise Exception(FORCED_OUTPUT_HANDLER_WITH_DEFAULT_PROMPT_MESSAGE)
             prompt = hub.pull("hwchase17/react")
 
         if llm is None:
@@ -97,7 +98,6 @@ def agent_factory(
             name=name,
             agent_factory=agent_factory,
             tools=tools,
-            output_handler=output_handler,
             chat_history=chat_history,
             runnable_config=runnable_config,
             verbose=verbose,