Overview

TogetherAIChatGenerator supports models hosted on Together AI, such as meta-llama/Llama-3.3-70B-Instruct-Turbo. For the full list of supported models, see Together AI documentation.

This component needs a list of ChatMessage objects to operate. ChatMessage is a data class that contains a message, a role (who generated the message, such as user, assistant, system, function), and optional metadata.

You can pass any text generation parameters valid for the Together AI chat completion API directly to this component using the generation_kwargs parameter in __init__ or the generation_kwargs parameter in run method. For more details on the parameters supported by the Together AI API, see Together AI API documentation.

TogetherAIChatGenerator can support function calling with Together AI models. It is fully compatible with Haystack Tool and Toolset for function-calling capabilities with supported models.

To use this integration, you need to have an active TogetherAI subscription with sufficient credits and an API key. You can provide it with:

• The TOGETHER_API_KEY environment variable (recommended)
• The api_key init parameter and Haystack Secret API: Secret.from_token("your-api-key-here")

By default the component targets Together AI’s OpenAI-compatible base URL https://api.together.xyz/v1, which you can override with api_base_url if needed.

Streaming

TogetherAIChatGenerator supports streaming responses from the LLM, allowing tokens to be emitted as they are generated. To enable streaming, pass a callable to the streaming_callback parameter during initialization.

Usage

Install the togetherai-haystack package to use the TogetherAIChatGenerator:

pip install togetherai-haystack

On its own

from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.togetherai import TogetherAIChatGenerator

client = TogetherAIChatGenerator()
response = client.run(
    [ChatMessage.from_user("What are Agentic Pipelines? Be brief.")]
)
print(response["replies"][0].text)

With streaming:

from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.togetherai import TogetherAIChatGenerator

client = TogetherAIChatGenerator(
    model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
    streaming_callback=lambda chunk: print(chunk.content, end="", flush=True),
)

response = client.run(
    [ChatMessage.from_user("What are Agentic Pipelines? Be brief.")]
)

# check the model used for the response
print("\n\nModel used:", response["replies"][0].meta.get("model"))

In a pipeline

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.togetherai import TogetherAIChatGenerator

prompt_builder = ChatPromptBuilder()
llm = TogetherAIChatGenerator(model="meta-llama/Llama-3.3-70B-Instruct-Turbo")

pipe = Pipeline()
pipe.add_component("builder", prompt_builder)
pipe.add_component("llm", llm)
pipe.connect("builder.prompt", "llm.messages")

messages = [
    ChatMessage.from_system("Give brief answers."),
    ChatMessage.from_user("Tell me about {{city}}"),
]

response = pipe.run(
    data={"builder": {"template": messages,
                      "template_variables": {"city": "Berlin"}}}
)
print(response)