DocumentationAPI Reference📓 Tutorials🧑‍🍳 Cookbook🤝 Integrations💜 Discord


This component enables chat completion using Cohere's large language models (LLMs).

Folder Path
Most common Position in a PipelineAfter a DynamicChatPromptBuilder
Mandatory Input variables“messages” a List of ChatMessage instances
Output variables"replies": a list of ChatMessage objects

”meta”: a list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on

This integration supports Cohere chat models such as command and command-light. Check out the most recent full list in Cohere documentation.

Parameters Overview

CohereChatGenerator needs a Cohere API key to work. You can write this key in:

  • The api_key init parameter
  • The COHERE_API_KEY environment variable (recommended)

Then, the component needs a prompt to operate, but you can pass any text generation parameters valid for the method directly to this component using the generation_kwargs parameter, both at initialization and to run() method. For more details on the parameters supported by the Cohere API, refer to the Cohere documentation.

Finally, the component needs a list of ChatMessage objects to operate. ChatMessage is a data class that contains a message, a role (who generated the message, such as user, assistant, system, function), and optional metadata.


CohereChatGenerator supports streaming the tokens from the LLM directly in output. To do so, pass a function to the streaming_callback init parameter.


You need to install cohere-haystack package to use the CohereChatGenerator:

pip install cohere-haystack

On its own

from haystack_integrations.components.generators.cohere import CohereChatGenerator
from haystack.dataclasses import ChatMessage

generator = CohereChatGenerator(api_key="<YOUR_COHERE_KEY>")
message = ChatMessage.from_user("What's Natural Language Processing? Be brief.")

In a Pipeline

You can also use CohereChatGenerator to use cohere chat models in your pipeline.

from haystack import Pipeline
from import DynamicChatPromptBuilder
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.cohere import CohereChatGenerator
from haystack.utils import Secret

pipe = Pipeline()
pipe.add_component("prompt_builder", DynamicChatPromptBuilder())
pipe.add_component("llm", CohereChatGenerator(Secret.from_token(COHERE_API_KEY)))
pipe.connect("prompt_builder", "llm")

country = "Germany"
system_message = ChatMessage.from_system("You are an assistant giving out valuable information to language learners.")
messages = [system_message, ChatMessage.from_user("What's the official language of {{ country }}?")]

res ={"prompt_builder": {"template_variables": {"country": country}, "prompt_source": messages}})

Related Links

Check out the API reference in the GitHub repo or in our docs: