Documentation

Agents

This page explains how to create an AI agent in Haystack capable of retrieving information, generating responses, and taking actions using various Haystack components.

Suggest Edits

What’s an AI Agent?

An AI agent is a system that can:

  • Understand user input (text, image, audio, and other queries),
  • Retrieve relevant information (documents or structured data),
  • Generate intelligent responses (using LLMs like OpenAI or Hugging Face models),
  • Perform actions (calling APIs, fetching live data, executing functions).

Understanding AI Agents

AI agents are autonomous systems that use large language models (LLMs) to make decisions and solve complex tasks. They interact with their environment using tools, memory, and reasoning.

What Makes an AI Agent

An AI agent is more than a chatbot. It actively plans, chooses the right tools and executes tasks to achieve a goal. Unlike traditional software, it adapts to new information and refines its process as needed.

  1. LLM as the Brain: The agent's core is an LLM, which understands context, processes natural language and serves as the central intelligence system.
  2. Tools for Interaction: Agents connect to external tools, APIs, and databases to gather information and take action.
  3. Memory for Context: Short-term memory helps track conversations, while long-term memory stores knowledge for future interactions.
  4. Reasoning and Planning: Agents break down complex problems, come up with step-by-step action plans, and adapt based on new data and feedback.

How AI Agents Work

An AI agent starts with a prompt that defines its role and objectives. It decides when to use tools, gathers data, and refines its approach through loops of reasoning and action. It evaluates progress and adjusts its strategy to improve results.

For example, a customer service agent answers queries using a database. If it lacks an answer, it fetches real-time data, summarizes it, and provides a response. A coding assistant understands project requirements, suggests solutions, and even writes code.

Key Components

You can build an AI agent in Haystack using the three main elements in a pipeline:

  • Chat Generators to generate tool calls (with tool name and arguments) or assistant responses with an LLM,
  • Tool class that allows the LLM to perform actions such as running a pipeline or calling an external API, connecting to the external world,
  • ToolInvoker component to execute tool calls generated by an LLM. It parses the LLM's tool-calling responses and invokes the appropriate tool with the correct arguments from the pipeline.

There are three ways of creating a tool in Haystack:

  • Tool class – Creates a tool representation for a consistent tool-calling experience across all Generators. It allows for most customization, as you can define its own name and description.
  • ComponentTool class – Wraps a Haystack component as a callable tool.
  • @tool decorator – Creates tools from Python functions and automatically uses their function name and docstring.

📘

Experimental Agent Component

Alternatively, it’s possible to use the Agent component from haystack-experimental package to build an agent that implements a tool-calling functionality with provider-agnostic chat model support. You can use Agent either as a standalone component or as a part of a pipeline.

Example Agents

Pipeline With Tools

Here’s an example of how you would build a tool-calling agent with the help of ToolInvoker.

This is what’s happening in this code example:

  1. OpenAIChatGenerator uses an LLM to analyze the user's message and determines whether to provide an assistant response or initiate a tool call.
  2. ConditionalRouter directs the output from the OpenAIChatGenerator to there_are_tool_calls branch if it’s a tool call or to final_replies to return to the user directly.
  3. ToolInvoker executes the tool call generated by the LLM. ComponentTool wraps the SerperDevWebSearch component that fetches real-time search results, making it accessible for ToolInvoker to execute it as a tool.
  4. After the tool provides its output, the ToolInvoker sends this information back to the OpenAIChatGenerator, along with the original user question stored by the MessageCollector.
from haystack.dataclasses import ChatMessage
from haystack.components.tools import ToolInvoker
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.components.routers import ConditionalRouter
from haystack.tools import ComponentTool
from haystack.components.websearch import SerperDevWebSearch
from haystack import Pipeline
from typing import Any, Dict, List
from haystack import component
from haystack.core.component.types import Variadic

# helper component to temporarily store last user query before the tool call 
@component()
class MessageCollector:
    def __init__(self):
        self._messages = []

    @component.output_types(messages=List[ChatMessage])
    def run(self, messages: Variadic[List[ChatMessage]]) -> Dict[str, Any]:

        self._messages.extend([msg for inner in messages for msg in inner])
        return {"messages": self._messages}

    def clear(self):
        self._messages = []

# Create a tool from a component
web_tool = ComponentTool(
    component=SerperDevWebSearch(top_k=3)
)

# Define routing conditions
routes = [
    {
        "condition": "{{replies[0].tool_calls | length > 0}}",
        "output": "{{replies}}",
        "output_name": "there_are_tool_calls",
        "output_type": List[ChatMessage],
    },
    {
        "condition": "{{replies[0].tool_calls | length == 0}}",
        "output": "{{replies}}",
        "output_name": "final_replies",
        "output_type": List[ChatMessage], 
    },
]

# Create the pipeline
tool_agent = Pipeline()
tool_agent.add_component("message_collector", MessageCollector())
tool_agent.add_component("generator", OpenAIChatGenerator(model="gpt-4o-mini", tools=[web_tool]))
tool_agent.add_component("router", ConditionalRouter(routes, unsafe=True))
tool_agent.add_component("tool_invoker", ToolInvoker(tools=[web_tool]))

tool_agent.connect("generator.replies", "router")
tool_agent.connect("router.there_are_tool_calls", "tool_invoker")
tool_agent.connect("router.there_are_tool_calls", "message_collector")
tool_agent.connect("tool_invoker.tool_messages", "message_collector")
tool_agent.connect("message_collector", "generator.messages")

messages = [
    ChatMessage.from_system("You're a helpful agent choosing the right tool when necessary"), 
    ChatMessage.from_user("How is the weather in Berlin?")]
result = tool_agent.run({"messages": messages})

print(result["router"]["final_replies"][0].text)

Resulting in:

>>> The current weather in Berlin is around 46°F (8°C) with cloudy conditions. The high for today is forecasted to reach 48°F (9°C) and the low is expected to be around 37°F (3°C). The humidity is quite high at 92%, and there is a light wind blowing at 4 mph.

For more detailed weather updates, you can check the following links:
- [AccuWeather](https://www.accuweather.com/en/de/berlin/10178/weather-forecast/178087)
- [Weather.com](https://weather.com/weather/today/l/5ca23443513a0fdc1d37ae2ffaf5586162c6fe592a66acc9320a0d0536be1bb9)

Tool-Calling Agent

You can create a similar tool-calling agent with the experimental Agent component:

from haystack_experimental.components.agents import Agent
from haystack_experimental.tools.component_tool import ComponentTool
from haystack.components.websearch import SerperDevWebSearch
from haystack.dataclasses import Document

def web_doc_handler(documents) -> str:
    result_str = "Here are the documents retrieved from web:\n"
    for document in documents:
        result_str += document.content + "\n"

    return result_str

web_tool = ComponentTool(
    component=SerperDevWebSearch(top_k=3),
    outputs={
        "message": {"source": "documents", "handler": web_doc_handler},
        "documents": {"source": "documents"},
    },
)

tool_calling_agent = Agent(
    chat_generator=OpenAIChatGenerator(model="gpt-4o-mini"),
    system_prompt="You're a helpful agent choosing the right tool when necessary",
    tools=[web_tool],
    exit_condition="text",
    state_schema={"documents": {"type": List[Document]}},
)

user_message = ChatMessage.from_user("How is the weather in Berlin?")
result = tool_calling_agent.run(messages=[user_message])

print(result["messages"][-1].text)

Resulting in:

>>> The current weather in Berlin is approximately 47°F (about 8°C) and cloudy. The high for today is expected to be 48°F (9°C) with a low of 37°F (3°C). There is rain and drizzle, with the humidity at 92% and winds at 4 mph.

Updated 3 days ago