
Most common position in a pipeline	After a ChatPromptBuilder
Mandatory init variables	"api_key": The IBM Cloud API key. Can be set with `WATSONX_API_KEY` env var. "project_id": The IBM Cloud project ID. Can be set with `WATSONX_PROJECT_ID` env var.
Mandatory run variables	"messages" A list of `ChatMessage` objects
Output variables	"replies": A list of `ChatMessage` objects
API reference	Watsonx
GitHub link	https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/watsonx

This integration supports IBM watsonx.ai foundation models such as ibm/granite-13b-chat-v2, ibm/llama-2-70b-chat, ibm/llama-3-70b-instruct, and similar. These models provide high-quality chat completion capabilities through IBM's cloud platform. Check out the most recent full list in the IBM watsonx.ai documentation.

Overview

WatsonxChatGenerator needs IBM Cloud credentials to work. You can set these in:

The api_key and project_id init parameters using Secret API
The WATSONX_API_KEY and WATSONX_PROJECT_ID environment variables (recommended)

Then, the component needs a prompt to operate, but you can pass any text generation parameters valid for the IBM watsonx.ai API directly to this component using the generation_kwargs parameter, both at initialization and to run() method. For more details on the parameters supported by the IBM watsonx.ai API, refer to the IBM watsonx.ai documentation.

Finally, the component needs a list of ChatMessage objects to operate. ChatMessage is a data class that contains a message, a role (who generated the message, such as user, assistant, system, function), and optional metadata.

Streaming

This Generator supports streaming the tokens from the LLM directly in output. To do so, pass a function to the streaming_callback init parameter.

Usage

You need to install watsonx-haystack package to use the WatsonxChatGenerator:

pip install watsonx-haystack

On its own

from haystack_integrations.components.generators.watsonx.chat.chat_generator import WatsonxChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret

generator = WatsonxChatGenerator(
    api_key=Secret.from_env_var("WATSONX_API_KEY"),
    project_id=Secret.from_env_var("WATSONX_PROJECT_ID"),
    model="ibm/granite-13b-instruct-v2"
)

message = ChatMessage.from_user("What's Natural Language Processing? Be brief.")
print(generator.run([message]))

In a Pipeline

You can also use WatsonxChatGenerator to use IBM watsonx.ai chat models in your pipeline.

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.watsonx.chat.chat_generator import WatsonxChatGenerator
from haystack.utils import Secret

pipe = Pipeline()
pipe.add_component("prompt_builder", ChatPromptBuilder())
pipe.add_component("llm", WatsonxChatGenerator(
    api_key=Secret.from_env_var("WATSONX_API_KEY"),
    project_id=Secret.from_env_var("WATSONX_PROJECT_ID"),
    model="ibm/granite-13b-instruct-v2"
))
pipe.connect("prompt_builder", "llm")

country = "Germany"
system_message = ChatMessage.from_system("You are an assistant giving out valuable information to language learners.")
messages = [system_message, ChatMessage.from_user("What's the official language of {{ country }}?")]

res = pipe.run(data={"prompt_builder": {"template_variables": {"country": country}, "template": messages}})
print(res)