STACKITChatGenerator
This component enables chat completions using the STACKIT API.
| Most common position in a pipeline | After a ChatPromptBuilder |
| Mandatory init variables | "model": The model used through the STACKIT API |
| Mandatory run variables | “messages”: A list of ChatMessage objects |
| Output variables | "replies": A list of ChatMessage objects”meta”: A list of dictionaries with the metadata associated with each reply (such as token count, finish reason, and so on) |
| API reference | STACKIT |
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/stackit |
Overview
STACKITChatGenerator enables text generation models served by STACKIT through their API.
Parameters
To use the STACKITChatGenerator, ensure you have set a STACKIT_API_KEY as an environment variable. Alternatively, provide the API key as another environment variable or a token by setting
api_key and using Haystack’s secret management.
Set your preferred supported model with the model parameter when initializing the component. See the full list of all supported models on the STACKIT website.
Optionally, you can change the default api_base_url, which is "https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1".
You can pass any text generation parameters valid for the STACKIT Chat Completion API directly to this component with the generation_kwargs parameter in the init or run methods.
The component needs a list of ChatMessage objects to run. ChatMessage is a data class that contains a message, a role (who generated the message, such as user, assistant, system, function), and optional metadata. Find out more about it ChatMessage documentation.
Streaming
This ChatGenerator supports streaming the tokens from the LLM directly into the output. To do so, pass a function to the streaming_callback init parameter.
Usage
Install the stackit-haystack package to use the STACKITChatGenerator:
pip install stackit-haystack
On its own
from haystack_integrations.components.generators.stackit import STACKITChatGenerator
from haystack.dataclasses import ChatMessage
generator = STACKITChatGenerator(model="neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8")
result = generator.run([ChatMessage.from_user("Tell me a joke.")])
print(result)
In a pipeline
You can also use STACKITChatGenerator in your pipeline.
from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.stackit import STACKITChatGenerator
prompt_builder = ChatPromptBuilder()
llm = STACKITChatGenerator(model="neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8")
messages = [ChatMessage.from_user("Question: {{question}} \\n")]
pipeline = Pipeline()
pipeline.add_component("prompt_builder", prompt_builder)
pipeline.add_component("llm", llm)
pipeline.connect("prompt_builder.prompt", "llm.messages")
result = pipeline.run({"prompt_builder": {"template_variables": {"question": "Tell me a joke."}, "template": messages}})
print(result)
For an example of streaming in a pipeline, refer to the examples in the STACKIT integration repository and on its dedicated integration page.
Updated 8 months ago
