OrcaRouterChatGenerator

This component enables chat completion through OrcaRouter, an OpenAI-compatible model routing gateway.

Most common position in a pipeline	After a ChatPromptBuilder
Mandatory init variables	api_key: An OrcaRouter API key. Can be set with ORCAROUTER_API_KEY env variable or passed to init() method.
Mandatory run variables	messages: A list of ChatMessage objects
Output variables	replies: A list of ChatMessage objects
API reference	OrcaRouter
GitHub link	https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/orcarouter
Package name	orcarouter-haystack

Overview

The OrcaRouterChatGenerator enables you to use models from multiple providers (such as openai/gpt-4o-mini, anthropic/claude-opus-4.8, and google/gemini-2.5-flash) by making chat completion calls to the OrcaRouter API. Models are addressed with a provider/model namespace, and you can browse the available models in the OrcaRouter model catalog.

This generator also supports OrcaRouter-specific features such as:

• Automatic routing with the orcarouter/auto model, which lets OrcaRouter pick a live upstream model per request based on the policy configured in your OrcaRouter console.
• Provider routing and model fallback that are configurable with the generation_kwargs parameter during initialization or runtime. OrcaRouter-specific routing options are forwarded to the gateway through extra_body.

This component uses the same ChatMessage format as other Haystack Chat Generators for structured input and output. For more information, see the ChatMessage documentation.

Tool Support

OrcaRouterChatGenerator supports function calling through the tools parameter, which accepts flexible tool configurations:

• A list of Tool objects: Pass individual tools as a list
• A single Toolset: Pass an entire Toolset directly
• Mixed Tools and Toolsets: Combine multiple Toolsets with standalone tools in a single list

This allows you to organize related tools into logical groups while also including standalone tools as needed.

python

from haystack.tools import Tool, Toolset
from haystack_integrations.components.generators.orcarouter import OrcaRouterChatGenerator

# Create individual tools
weather_tool = Tool(name="weather", description="Get weather info", ...)
news_tool = Tool(name="news", description="Get latest news", ...)

# Group related tools into a toolset
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])

# Pass mixed tools and toolsets to the generator
generator = OrcaRouterChatGenerator(
    tools=[math_toolset, weather_tool, news_tool]  # Mix of Toolset and Tool objects
)

For more details on working with tools, see the Tool and Toolset documentation.

Initialization

To use this integration, you need an OrcaRouter API key. You can provide it with the ORCAROUTER_API_KEY environment variable or by using a Secret.

Then, install the orcarouter-haystack integration:

shell

pip install orcarouter-haystack

Streaming

OrcaRouterChatGenerator supports streaming responses from the LLM, allowing tokens to be emitted as they are generated. To enable streaming, pass a callable to the streaming_callback parameter during initialization.

Usage

On its own

python

from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.orcarouter import (
    OrcaRouterChatGenerator,
)

client = OrcaRouterChatGenerator(model="openai/gpt-4o-mini")
response = client.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")])
print(response["replies"][0].text)

With automatic routing and streaming:

python

from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.orcarouter import (
    OrcaRouterChatGenerator,
)

client = OrcaRouterChatGenerator(
    model="orcarouter/auto",
    streaming_callback=lambda chunk: print(chunk.content, end="", flush=True),
)

response = client.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")])

# check the model used for the response
print("\n\n Model used: ", response["replies"][0].meta["model"])

With a fallback chain:

python

from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.orcarouter import (
    OrcaRouterChatGenerator,
)

client = OrcaRouterChatGenerator(
    model="openai/gpt-4o-mini",
    generation_kwargs={
        "extra_body": {
            "route": "fallback",
            "models": [
                "openai/gpt-4o-mini",
                "anthropic/claude-haiku-4.5",
                "google/gemini-2.5-flash",
            ],
        },
    },
)

response = client.run([ChatMessage.from_user("What is Haystack?")])
print(response["replies"][0].text)

In a pipeline

python

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.orcarouter import (
    OrcaRouterChatGenerator,
)

prompt_builder = ChatPromptBuilder()
llm = OrcaRouterChatGenerator(model="openai/gpt-4o-mini")

pipe = Pipeline()
pipe.add_component("builder", prompt_builder)
pipe.add_component("llm", llm)
pipe.connect("builder.prompt", "llm.messages")

messages = [
    ChatMessage.from_system("Give brief answers."),
    ChatMessage.from_user("Tell me about {{city}}"),
]

response = pipe.run(
    data={"builder": {"template": messages, "template_variables": {"city": "Berlin"}}},
)
print(response)