DocumentationAPI ReferenceπŸ““ TutorialsπŸ§‘β€πŸ³ Cookbook🀝 IntegrationsπŸ’œ Discord🎨 Studio
Documentation

OpenAIChatGenerator

OpenAIChatGenerator enables chat completion using OpenAI’s large language models (LLMs).

Most common position in a pipelineAfter a ChatPromptBuilder
Mandatory init variables"api_key": An OpenAI API key. Can be set with OPENAI_API_KEY env var.
Mandatory run variablesβ€œmessages”: A list of ChatMessage objects representing the chat
Output variablesβ€œreplies”: A list of alternative replies of the LLM to the input chat
API referenceGenerators
GitHub linkhttps://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/chat/openai.py

Overview

OpenAIChatGenerator supports OpenAI models starting from gpt-3.5-turbo and later (gpt-4, gpt-4-turbo, and so on).

OpenAIChatGenerator needs an OpenAI key to work. It uses an OPENAI_API_KEYΒ environment variable by default. Otherwise, you can pass an API key at initialization with api_key:

generator = OpenAIChatGenerator(model="gpt-4o-mini")

Then, the component needs a list of ChatMessage objects to operate. ChatMessage is a data class that contains a message, a role (who generated the message, such as user, assistant, system, function), and optional metadata. See the usage section for an example.

You can pass any chat completion parameters valid for the openai.ChatCompletion.create method directly to OpenAIChatGenerator using the generation_kwargs parameter, both at initialization and to run() method. For more details on the parameters supported by the OpenAI API, refer to the OpenAI documentation.

OpenAIChatGenerator can support custom deployments of your OpenAI models through the api_base_url init parameter.

Streaming

OpenAIChatGenerator supports streaming the tokens from the LLM directly in output. To do so, pass a function to the streaming_callback init parameter. Note that streaming the tokens is only compatible with generating a single response, so n must be set to 1 for streaming to work.

πŸ“˜

This component is designed for chat completion, so it expects a list of messages, not a single string. If you want to use OpenAI LLMs for text generation (such as translation or summarization tasks) or don’t want to use the ChatMessage object, use OpenAIGenerator instead.

Usage

On its own

Basic usage:

from haystack.dataclasses import ChatMessage
from haystack.components.generators.chat import OpenAIChatGenerator

client = OpenAIChatGenerator()
response = client.run(
	  [ChatMessage.from_user("What's Natural Language Processing? Be brief.")]
)
print(response)

>> {'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, _content=
>> [TextContent(text='Natural Language Processing (NLP) is a field of artificial
>> intelligence that focuses on the interaction between computers and humans through
>> natural language. It involves enabling machines to understand, interpret, and
>> generate human language in a meaningful way, facilitating tasks such as 
>> language translation, sentiment analysis, and text summarization.')], 
>> _name=None, _meta={'model': 'gpt-4o-mini-2024-07-18', 'index': 0, 
>> 'finish_reason': 'stop', 'usage': {'completion_tokens': 59, 'prompt_tokens': 15,
>>  'total_tokens': 74, 'completion_tokens_details': {'accepted_prediction_tokens': 
>>  0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 
>>  'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}})]}

With streaming:

from haystack.dataclasses import ChatMessage
from haystack.components.generators.chat import OpenAIChatGenerator

client = OpenAIChatGenerator(streaming_callback=lambda chunk: print(chunk.content, end="", flush=True))
response = client.run(
	  [ChatMessage.from_user("What's Natural Language Processing? Be brief.")]
)
print(response)

>> Natural Language Processing (NLP) is a field of artificial intelligence that 
>> focuses on the interaction between computers and humans through natural language. 
>> It involves enabling machines to understand, interpret, and generate human 
>> language in a way that is both meaningful and useful. NLP encompasses various 
>> tasks, including speech recognition, language translation, sentiment analysis, 
>> and text summarization.{'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 
>> 'assistant'>, _content=[TextContent(text='Natural Language Processing (NLP) is a 
>> field of artificial intelligence that focuses on the interaction between computers 
>> and humans through natural language. It involves enabling machines to understand, 
>> interpret, and generate human language in a way that is both meaningful and 
>> useful. NLP encompasses various tasks, including speech recognition, language 
>> translation, sentiment analysis, and text summarization.')], _name=None, _meta={'
>> model': 'gpt-4o-mini-2024-07-18', 'index': 0, 'finish_reason': 'stop', 
>> 'completion_start_time': '2025-05-15T13:32:16.572912', 'usage': None})]}

In a Pipeline

from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack import Pipeline
from haystack.utils import Secret

# no parameter init, we don't use any runtime template variables
prompt_builder = ChatPromptBuilder()
llm = OpenAIChatGenerator(api_key=Secret.from_env_var("OPENAI_API_KEY"), model="gpt-4o-mini")

pipe = Pipeline()
pipe.add_component("prompt_builder", prompt_builder)
pipe.add_component("llm", llm)
pipe.connect("prompt_builder.prompt", "llm.messages")
location = "Berlin"
messages = [ChatMessage.from_system("Always respond in German even if some input data is in other languages."),
            ChatMessage.from_user("Tell me about {{location}}")]
pipe.run(data={"prompt_builder": {"template_variables":{"location": location}, "template": messages}})

>> {'llm': {'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, 
>> _content=[TextContent(text='Berlin ist die Hauptstadt Deutschlands und eine der 
>> bedeutendsten StΓ€dte Europas. Sie ist bekannt fΓΌr ihre reiche Geschichte, 
>> kulturelle Vielfalt und kreative Szene. \n\nDie Stadt hat eine bewegte 
>> Vergangenheit, die stark von der Teilung zwischen Ost- und Westberlin wΓ€hrend 
>> des Kalten Krieges geprΓ€gt war. Die Berliner Mauer, die von 1961 bis 1989 die 
>> Stadt teilte, ist heute ein Symbol fΓΌr die Wiedervereinigung und die Freiheit. 
>> \n\nBerlin bietet eine FΓΌlle von SehenswΓΌrdigkeiten, darunter das Brandenburger 
>> Tor, den Reichstag, die Museumsinsel und den Alexanderplatz. Die Stadt ist auch 
>> fΓΌr ihre lebendige Kunst- und Musikszene bekannt, mit zahlreichen Galerien, 
>> Theatern und Clubs. ')], _name=None, _meta={'model': 'gpt-4o-mini-2024-07-18', 
>> 'index': 0, 'finish_reason': 'stop', 'usage': {'completion_tokens': 260, 
>> 'prompt_tokens': 29, 'total_tokens': 289, 'completion_tokens_details': 
>> {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 
>> 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 
>> 'cached_tokens': 0}}})]}}

Additional References

πŸ““ Tutorial: Building a Chat Application with Function Calling

πŸ§‘β€πŸ³ Cookbook: Function Calling with OpenAIChatGenerator