Function Calling

Learn about function calling and how to use it in Haystack.

Function calling is a powerful feature that significantly enhances the capabilities of Large Language Models (LLMs). It enables better functionality, immediate data access, and interaction, and sets up for integration with external APIs and services. Function calling turns LLMs into adaptable tools for various use case scenarios.

Use Cases

Function calling is useful for a variety of purposes, but two main points are particularly notable:

1. Enhanced LLM Functionality: Function calling enhances the capabilities of LLMs beyond just text generation. It allows to convert human-generated prompts into precise function invocation descriptors. These descriptors can then be used by connected LLM frameworks to perform computations, manipulate data, and interact with external APIs. This expansion of functionality makes LLMs adaptable tools for a wide array of tasks and industries.
2. Real-Time Data Access and Interaction: Function calling lets LLMs create function calls that access and interact with real-time data. This is necessary for apps that need current data, like news, weather, or financial market updates. By giving access to the latest information, this feature greatly improves the usefulness and trustworthiness of LLMs in changing and time-critical situations.

Important Note

The model doesn't actually call the function. Function calling returns the name of a function and the arguments to invoke it. The actual invocation is performed by your code (or by a Haystack component such as ToolInvoker or Agent).

Example

Let's walk through function calling in Haystack in three steps: define a tool, let the LLM pick it, and then actually invoke it.

Real-world usage

We split function calling into separate steps below for clarity. In real applications, the Agent component handles the full loop for you. See Using the Agent component at the end of this section.

1. Define a Tool

The simplest way to expose a Python function to an LLM is the @tool decorator. Type hints (and Annotated metadata) are used to automatically build the JSON schema the LLM needs.

python

from typing import Annotated, Literal
from haystack.tools import tool


@tool
def get_weather(
    city: Annotated[str, "The city to get the weather for"],
    unit: Annotated[Literal["celsius", "fahrenheit"], "Temperature unit"] = "celsius",
):
    """Get the current weather for a city."""
    # In a real application, this would call a weather API.
    return {"city": city, "temperature": 18, "unit": unit, "condition": "Partly Cloudy"}

This produces a Tool instance you can pass directly to a ChatGenerator. If you prefer to build it explicitly, you can also instantiate Tool(...) yourself.

2. Let the LLM Pick the Tool

Pass the tool to a ChatGenerator and run it on a user message. The model decides whether to call the tool and, if so, with which arguments. The result lives on replies[0].tool_calls.

python

from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

generator = OpenAIChatGenerator(tools=[get_weather])
response = generator.run(
    messages=[ChatMessage.from_user("What's the weather like in Berlin?")],
)

print(response["replies"][0].tool_calls)
# >> [ToolCall(tool_name='get_weather', arguments={'city': 'Berlin', 'unit': 'celsius'}, id='call_...')]

At this point the model has only requested a call. Nothing has been executed yet.

3. Actually Invoke the Tool

To execute the requested tool calls, use the ToolInvoker component. It takes a list of ChatMessage objects containing tool calls, runs the corresponding tools, and returns new ChatMessage objects (with role tool) carrying the results as ToolCallResults. To get a final natural-language answer, feed those results back to the ChatGenerator.

python

from haystack.components.tools import ToolInvoker

invoker = ToolInvoker(tools=[get_weather])

# Run the tools requested by the assistant.
tool_messages = invoker.run(messages=response["replies"])["tool_messages"]

# Send the full conversation back to the model for the final reply.
messages = [
    ChatMessage.from_user("What's the weather like in Berlin?"),
    *response["replies"],  # assistant message with the tool call
    *tool_messages,  # tool result messages
]
final = generator.run(messages=messages)
print(final["replies"][0].text)
# The weather in Berlin is partly cloudy with a temperature of 18°C.

Using the Agent component

In real applications, you typically don't manage the loop yourself: the Agent component does it for you. Internally, it wraps a ChatGenerator and a ToolInvoker and iterates until the model produces a final answer.

python

from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

agent = Agent(
    chat_generator=OpenAIChatGenerator(),
    tools=[get_weather],
)

response = agent.run(
    messages=[ChatMessage.from_user("What's the weather like in Berlin?")],
)
print(response["messages"][-1].text)
# The weather in Berlin is partly cloudy with a temperature of 18°C.

Additional References

• Tool, Toolset, and ComponentTool cover the building blocks for defining tools, grouping them, and exposing existing Haystack components as tools.
• To connect an LLM to external services or other Haystack applications, the recommended approach is the Model Context Protocol (MCP). Haystack provides MCPTool and MCPToolset for connecting to MCP servers over Streamable HTTP or stdio. Install them with pip install mcp-haystack.

📓 Tutorial: Building a Chat Application with Function Calling