
Most common position in a pipeline	After a PromptBuilder
Mandatory init variables	"api_key": An IBM Cloud API key. Can be set with `WATSONX_API_KEY` env var. "project_id": An IBM Cloud project ID. Can be set with `WATSONX_PROJECT_ID` env var.
Mandatory run variables	"prompt": A string containing the prompt for the LLM
Output variables	"replies": A list of strings with all the replies generated by the LLM "meta": A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on
API reference	Watsonx
GitHub link	https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/watsonx

Overview

This integration supports IBM watsonx.ai foundation models such as ibm/granite-13b-chat-v2, ibm/llama-2-70b-chat, ibm/llama-3-70b-instruct, and similar. These models provide high-quality text generation capabilities through IBM's cloud platform. Check out the most recent full list in the IBM watsonx.ai documentation.

Parameters

WatsonxGenerator needs IBM Cloud credentials to work. You can provide these in:

The WATSONX_API_KEY environment variable (recommended)
The WATSONX_PROJECT_ID environment variable (recommended)
The api_key and project_id init parameters using Haystack Secret API: Secret.from_token("your-api-key-here")

Set your preferred IBM watsonx.ai model in the model parameter when initializing the component. The default model is ibm/granite-3-2b-instruct.

WatsonxGenerator requires a prompt to generate text, but you can pass any text generation parameters available in the IBM watsonx.ai API directly to this component using the generation_kwargs parameter, both at initialization and to run() method. For more details on the parameters supported by the IBM watsonx.ai API, see IBM watsonx.ai documentation.

The component also supports system prompts that can be set at initialization or passed during runtime to provide context or instructions for the generation.

Finally, the component run method requires a single string prompt to generate text.

Streaming

This Generator supports streaming the tokens from the LLM directly in output. To do so, pass a function to the streaming_callback init parameter.

Usage

Install the watsonx-haystack package to use the WatsonxGenerator:

pip install watsonx-haystack

On its own

from haystack_integrations.components.generators.watsonx.generator import WatsonxGenerator
from haystack.utils import Secret

generator = WatsonxGenerator(
    api_key=Secret.from_env_var("WATSONX_API_KEY"),
    project_id=Secret.from_env_var("WATSONX_PROJECT_ID")
)

print(generator.run("What's Natural Language Processing? Be brief."))

In a pipeline

You can also use WatsonxGenerator with the IBM watsonx.ai models in your pipeline.

from haystack import Pipeline
from haystack.components.builders import PromptBuilder
from haystack_integrations.components.generators.watsonx.generator import WatsonxGenerator
from haystack.utils import Secret

template = """
You are an assistant giving out valuable information to language learners.
Answer this question, be brief.

Question: {{ query }}?
"""

pipe = Pipeline()
pipe.add_component("prompt_builder", PromptBuilder(template))
pipe.add_component("llm", WatsonxGenerator(
    api_key=Secret.from_env_var("WATSONX_API_KEY"),
    project_id=Secret.from_env_var("WATSONX_PROJECT_ID")
))
pipe.connect("prompt_builder", "llm")

query = "What language is spoken in Germany?"
res = pipe.run(data={"prompt_builder": {"query": query}})

print(res)