Overview

STACKITChatGenerator enables text generation models served by STACKIT through their API.

Parameters

To use the STACKITChatGenerator, ensure you have set a STACKIT_API_KEY as an environment variable. Alternatively, provide the API key as another environment variable or a token by setting
api_key and using Haystack’s secret management.

Set your preferred supported model with the model parameter when initializing the component. See the full list of all supported models on the STACKIT website.

Optionally, you can change the default api_base_url, which is "https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1".

You can pass any text generation parameters valid for the STACKIT Chat Completion API directly to this component with the generation_kwargs parameter in the init or run methods.

The component needs a list of ChatMessage objects to run. ChatMessage is a data class that contains a message, a role (who generated the message, such as user, assistant, system, function), and optional metadata. Find out more about it ChatMessage documentation.

Streaming

This ChatGenerator supports streaming the tokens from the LLM directly into the output. To do so, pass a function to the streaming_callback init parameter.

Usage

Install the stackit-haystack package to use the STACKITChatGenerator:

pip install stackit-haystack

On its own

from haystack_integrations.components.generators.stackit import STACKITChatGenerator
from haystack.dataclasses import ChatMessage

generator = STACKITChatGenerator(model="neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8")

result = generator.run([ChatMessage.from_user("Tell me a joke.")])
print(result)

In a pipeline

You can also use STACKITChatGenerator in your pipeline.

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.dataclasses import ChatMessage

from haystack_integrations.components.generators.stackit import STACKITChatGenerator

prompt_builder = ChatPromptBuilder()
llm = STACKITChatGenerator(model="neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8")

messages = [ChatMessage.from_user("Question: {{question}} \\n")]

pipeline = Pipeline()
pipeline.add_component("prompt_builder", prompt_builder)
pipeline.add_component("llm", llm)

pipeline.connect("prompt_builder.prompt", "llm.messages")

result = pipeline.run({"prompt_builder": {"template_variables": {"question": "Tell me a joke."}, "template": messages}})

print(result)

For an example of streaming in a pipeline, refer to the examples in the STACKIT integration repository and on its dedicated integration page.