Amazon Bedrock integration for Haystack
Module haystack_integrations.components.generators.amazon_bedrock.generator
AmazonBedrockGenerator
Generates text using models hosted on Amazon Bedrock.
For example, to use the Anthropic Claude model, pass 'anthropic.claude-v2' in the model
parameter.
Provide AWS credentials either through the local AWS profile or directly through
aws_access_key_id
, aws_secret_access_key
, aws_session_token
, and aws_region_name
parameters.
Usage example
from haystack_integrations.components.generators.amazon_bedrock import AmazonBedrockGenerator
generator = AmazonBedrockGenerator(
model="anthropic.claude-v2",
max_length=99
)
print(generator.run("Who is the best American actor?"))
AmazonBedrockGenerator uses AWS for authentication. You can use the AWS CLI to authenticate through your IAM.
For more information on setting up an IAM identity-based policy, see [Amazon Bedrock documentation]
(https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html).
If the AWS environment is configured correctly, the AWS credentials are not required as they're loaded
automatically from the environment or the AWS configuration file.
If the AWS environment is not configured, set aws_access_key_id
, aws_secret_access_key
,
aws_session_token
, and aws_region_name
as environment variables or pass them as
Secret arguments. Make sure the region you set
supports Amazon Bedrock.
AmazonBedrockGenerator.__init__
def __init__(
model: str,
aws_access_key_id: Optional[Secret] = Secret.from_env_var(
"AWS_ACCESS_KEY_ID", strict=False),
aws_secret_access_key: Optional[Secret] = Secret.
from_env_var( # noqa: B008
"AWS_SECRET_ACCESS_KEY", strict=False),
aws_session_token: Optional[Secret] = Secret.from_env_var(
"AWS_SESSION_TOKEN", strict=False),
aws_region_name: Optional[Secret] = Secret.from_env_var(
"AWS_DEFAULT_REGION", strict=False),
aws_profile_name: Optional[Secret] = Secret.from_env_var("AWS_PROFILE",
strict=False),
max_length: Optional[int] = 100,
truncate: Optional[bool] = True,
streaming_callback: Optional[Callable[[StreamingChunk], None]] = None,
boto3_config: Optional[Dict[str, Any]] = None,
model_family: Optional[MODEL_FAMILIES] = None,
**kwargs)
Create a new AmazonBedrockGenerator
instance.
Arguments:
model
: The name of the model to use.aws_access_key_id
: The AWS access key ID.aws_secret_access_key
: The AWS secret access key.aws_session_token
: The AWS session token.aws_region_name
: The AWS region name. Make sure the region you set supports Amazon Bedrock.aws_profile_name
: The AWS profile name.max_length
: The maximum length of the generated text.truncate
: Whether to truncate the prompt or not.streaming_callback
: A callback function that is called when a new token is received from the stream. The callback function accepts StreamingChunk as an argument.boto3_config
: The configuration for the boto3 client.model_family
: The model family to use. If not provided, the model adapter is selected based on the model name.kwargs
: Additional keyword arguments to be passed to the model. These arguments are specific to the model. You can find them in the model's documentation.
Raises:
ValueError
: If the model name is empty or None.AmazonBedrockConfigurationError
: If the AWS environment is not configured correctly or the model is not supported.
AmazonBedrockGenerator.run
@component.output_types(replies=List[str])
def run(prompt: str,
streaming_callback: Optional[Callable[[StreamingChunk], None]] = None,
generation_kwargs: Optional[Dict[str, Any]] = None)
Generates a list of string response to the given prompt.
Arguments:
prompt
: The prompt to generate a response for.streaming_callback
: A callback function that is called when a new token is received from the stream.generation_kwargs
: Additional keyword arguments passed to the generator.
Raises:
ValueError
: If the prompt is empty or None.AmazonBedrockInferenceError
: If the model cannot be invoked.
Returns:
A dictionary with the following keys:
replies
: A list of generated responses.
AmazonBedrockGenerator.get_model_adapter
@classmethod
def get_model_adapter(
cls,
model: str,
model_family: Optional[str] = None) -> Type[BedrockModelAdapter]
Gets the model adapter for the given model.
If model_family
is provided, the adapter for the model family is returned.
If model_family
is not provided, the adapter is auto-detected based on the model name.
Arguments:
model
: The model name.model_family
: The model family.
Raises:
AmazonBedrockConfigurationError
: If the model family is not supported or the model cannot be auto-detected.
Returns:
The model adapter class, or None if no adapter is found.
AmazonBedrockGenerator.to_dict
def to_dict() -> Dict[str, Any]
Serializes the component to a dictionary.
Returns:
Dictionary with serialized data.
AmazonBedrockGenerator.from_dict
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AmazonBedrockGenerator"
Deserializes the component from a dictionary.
Arguments:
data
: Dictionary to deserialize from.
Returns:
Deserialized component.
Module haystack_integrations.components.generators.amazon_bedrock.adapters
BedrockModelAdapter
Base class for Amazon Bedrock model adapters.
Each subclass of this class is designed to address the unique specificities of a particular LLM it adapts, focusing on preparing the requests and extracting the responses from the Amazon Bedrock hosted LLMs.
BedrockModelAdapter.prepare_body
@abstractmethod
def prepare_body(prompt: str, **inference_kwargs) -> Dict[str, Any]
Prepares the body for the Amazon Bedrock request.
Each subclass should implement this method to prepare the request body for the specific model.
Arguments:
prompt
: The prompt to be sent to the model.inference_kwargs
: Additional keyword arguments passed to the handler.
Returns:
A dictionary containing the body for the request.
BedrockModelAdapter.get_responses
def get_responses(response_body: Dict[str, Any]) -> List[str]
Extracts the responses from the Amazon Bedrock response.
Arguments:
response_body
: The response body from the Amazon Bedrock request.
Returns:
A list of responses.
BedrockModelAdapter.get_stream_responses
def get_stream_responses(
stream, streaming_callback: Callable[[StreamingChunk],
None]) -> List[str]
Extracts the responses from the Amazon Bedrock streaming response.
Arguments:
stream
: The streaming response from the Amazon Bedrock request.streaming_callback
: The handler for the streaming response.
Returns:
A list of string responses.
AnthropicClaudeAdapter
Adapter for the Anthropic Claude models.
AnthropicClaudeAdapter.prepare_body
def prepare_body(prompt: str, **inference_kwargs) -> Dict[str, Any]
Prepares the body for the Claude model
Arguments:
prompt
: The prompt to be sent to the model.inference_kwargs
: Additional keyword arguments passed to the handler.
Returns:
A dictionary with the following keys:
prompt
: The prompt to be sent to the model.- specified inference parameters.
MistralAdapter
Adapter for the Mistral models.
MistralAdapter.prepare_body
def prepare_body(prompt: str, **inference_kwargs) -> Dict[str, Any]
Prepares the body for the Mistral model
Arguments:
prompt
: The prompt to be sent to the model.inference_kwargs
: Additional keyword arguments passed to the handler.
Returns:
A dictionary with the following keys:
prompt
: The prompt to be sent to the model.- specified inference parameters.
CohereCommandAdapter
Adapter for the Cohere Command model.
CohereCommandAdapter.prepare_body
def prepare_body(prompt: str, **inference_kwargs) -> Dict[str, Any]
Prepares the body for the Command model
Arguments:
prompt
: The prompt to be sent to the model.inference_kwargs
: Additional keyword arguments passed to the handler.
Returns:
A dictionary with the following keys:
prompt
: The prompt to be sent to the model.- specified inference parameters.
CohereCommandRAdapter
Adapter for the Cohere Command R models.
CohereCommandRAdapter.prepare_body
def prepare_body(prompt: str, **inference_kwargs: Any) -> Dict[str, Any]
Prepares the body for the Command model
Arguments:
prompt
: The prompt to be sent to the model.inference_kwargs
: Additional keyword arguments passed to the handler.
Returns:
A dictionary with the following keys:
prompt
: The prompt to be sent to the model.- specified inference parameters.
AI21LabsJurassic2Adapter
Model adapter for AI21 Labs' Jurassic 2 models.
AI21LabsJurassic2Adapter.prepare_body
def prepare_body(prompt: str, **inference_kwargs) -> Dict[str, Any]
Prepares the body for the Jurassic 2 model.
Arguments:
prompt
: The prompt to be sent to the model.inference_kwargs
: Additional keyword arguments passed to the handler.
Returns:
A dictionary with the following keys:
prompt
: The prompt to be sent to the model.- specified inference parameters.
AmazonTitanAdapter
Adapter for Amazon's Titan models.
AmazonTitanAdapter.prepare_body
def prepare_body(prompt: str, **inference_kwargs) -> Dict[str, Any]
Prepares the body for the Titan model
Arguments:
prompt
: The prompt to be sent to the model.inference_kwargs
: Additional keyword arguments passed to the handler.
Returns:
A dictionary with the following keys
inputText
: The prompt to be sent to the model.- specified inference parameters.
MetaLlamaAdapter
Adapter for Meta's Llama2 models.
MetaLlamaAdapter.prepare_body
def prepare_body(prompt: str, **inference_kwargs) -> Dict[str, Any]
Prepares the body for the Llama2 model
Arguments:
prompt
: The prompt to be sent to the model.inference_kwargs
: Additional keyword arguments passed to the handler.
Returns:
A dictionary with the following keys:
prompt
: The prompt to be sent to the model.- specified inference parameters.
Module haystack_integrations.common.amazon_bedrock.errors
AmazonBedrockError
Any error generated by the Amazon Bedrock integration.
This error wraps its source transparently in such a way that its attributes
can be accessed directly: for example, if the original error has a message
attribute,
AmazonBedrockError.message
will exist and have the expected content.
AWSConfigurationError
Exception raised when AWS is not configured correctly
AmazonBedrockConfigurationError
Exception raised when AmazonBedrock node is not configured correctly
AmazonBedrockInferenceError
Exception for issues that occur in the Bedrock inference node
Module haystack_integrations.components.generators.amazon_bedrock.handlers
DefaultPromptHandler
DefaultPromptHandler resizes the prompt to ensure that the prompt and answer token lengths together are within the model_max_length.
DefaultPromptHandler.__init__
def __init__(tokenizer: Union[str, PreTrainedTokenizerBase],
model_max_length: int,
max_length: int = 100)
Arguments:
tokenizer
: The tokenizer to be used to tokenize the prompt.model_max_length
: The maximum length of the prompt and answer tokens combined.max_length
: The maximum length of the answer tokens.
Raises:
ValueError
: If the tokenizer is not a string or aPreTrainedTokenizer
orPreTrainedTokenizerFast
instance.
DefaultPromptHandler.__call__
def __call__(prompt: str, **kwargs) -> Dict[str, Union[str, int]]
Resizes the prompt to ensure that the prompt and answer is within the model_max_length
Arguments:
prompt
: the prompt to be sent to the model.kwargs
: Additional keyword arguments passed to the handler.
Returns:
A dictionary containing the resized prompt and additional information.
Module haystack_integrations.components.generators.amazon_bedrock.chat.chat_generator
AmazonBedrockChatGenerator
Completes chats using LLMs hosted on Amazon Bedrock available via the Bedrock Converse API.
For example, to use the Anthropic Claude 3 Sonnet model, initialize this component with the 'anthropic.claude-3-5-sonnet-20240620-v1:0' model name.
Usage example
from haystack_integrations.components.generators.amazon_bedrock import AmazonBedrockChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.components.generators.utils import print_streaming_chunk
messages = [ChatMessage.from_system("\nYou are a helpful, respectful and honest assistant, answer in German only"),
ChatMessage.from_user("What's Natural Language Processing?")]
client = AmazonBedrockChatGenerator(model="anthropic.claude-3-5-sonnet-20240620-v1:0",
streaming_callback=print_streaming_chunk)
client.run(messages, generation_kwargs={"max_tokens": 512})
AmazonBedrockChatGenerator uses AWS for authentication. You can use the AWS CLI to authenticate through your IAM. For more information on setting up an IAM identity-based policy, see [Amazon Bedrock documentation] (https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html).
If the AWS environment is configured correctly, the AWS credentials are not required as they're loaded
automatically from the environment or the AWS configuration file.
If the AWS environment is not configured, set aws_access_key_id
, aws_secret_access_key
,
and aws_region_name
as environment variables or pass them as
Secret arguments. Make sure the region you set
supports Amazon Bedrock.
AmazonBedrockChatGenerator.__init__
def __init__(
model: str,
aws_access_key_id: Optional[Secret] = Secret.from_env_var(
["AWS_ACCESS_KEY_ID"], strict=False),
aws_secret_access_key: Optional[Secret] = Secret.
from_env_var( # noqa: B008
["AWS_SECRET_ACCESS_KEY"], strict=False),
aws_session_token: Optional[Secret] = Secret.from_env_var(
["AWS_SESSION_TOKEN"], strict=False),
aws_region_name: Optional[Secret] = Secret.from_env_var(
["AWS_DEFAULT_REGION"], strict=False),
aws_profile_name: Optional[Secret] = Secret.from_env_var(
["AWS_PROFILE"], strict=False),
generation_kwargs: Optional[Dict[str, Any]] = None,
stop_words: Optional[List[str]] = None,
streaming_callback: Optional[Callable[[StreamingChunk], None]] = None,
boto3_config: Optional[Dict[str, Any]] = None)
Initializes the AmazonBedrockChatGenerator
with the provided parameters. The parameters are passed to the
Amazon Bedrock client.
Note that the AWS credentials are not required if the AWS environment is configured correctly. These are loaded
automatically from the environment or the AWS configuration file and do not need to be provided explicitly via
the constructor. If the AWS environment is not configured users need to provide the AWS credentials via the
constructor. Aside from model, three required parameters are aws_access_key_id
, aws_secret_access_key
,
and aws_region_name
.
Arguments:
model
: The model to use for text generation. The model must be available in Amazon Bedrock and must be specified in the format outlined in the Amazon Bedrock documentation.aws_access_key_id
: AWS access key ID.aws_secret_access_key
: AWS secret access key.aws_session_token
: AWS session token.aws_region_name
: AWS region name. Make sure the region you set supports Amazon Bedrock.aws_profile_name
: AWS profile name.generation_kwargs
: Keyword arguments sent to the model. These parameters are specific to a model. You can find them in the model's documentation. For example, you can find the Anthropic Claude generation parameters in Anthropic documentation.stop_words
: A list of stop words that stop the model from generating more text when encountered. You can provide them using this parameter or using the model'sgeneration_kwargs
under a model's specific key for stop words. For example, you can provide stop words for Anthropic Claude in thestop_sequences
key.streaming_callback
: A callback function called when a new token is received from the stream. By default, the model is not set up for streaming. To enable streaming, set this parameter to a callback function that handles the streaming chunks. The callback function receives a StreamingChunk object and switches the streaming mode on.boto3_config
: The configuration for the boto3 client.
Raises:
ValueError
: If the model name is empty or None.AmazonBedrockConfigurationError
: If the AWS environment is not configured correctly or the model is not supported.
AmazonBedrockChatGenerator.to_dict
def to_dict() -> Dict[str, Any]
Serializes the component to a dictionary.
Returns:
Dictionary with serialized data.
AmazonBedrockChatGenerator.from_dict
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AmazonBedrockChatGenerator"
Deserializes the component from a dictionary.
Arguments:
data
: Dictionary with serialized data.
Returns:
Instance of AmazonBedrockChatGenerator
.
Module haystack_integrations.components.embedders.amazon_bedrock.text_embedder
AmazonBedrockTextEmbedder
A component for embedding strings using Amazon Bedrock.
Usage example:
import os
from haystack_integrations.components.embedders.amazon_bedrock import AmazonBedrockTextEmbedder
os.environ["AWS_ACCESS_KEY_ID"] = "..."
os.environ["AWS_SECRET_ACCESS_KEY_ID"] = "..."
os.environ["AWS_REGION_NAME"] = "..."
embedder = AmazonBedrockTextEmbedder(
model="cohere.embed-english-v3",
input_type="search_query",
)
print(text_embedder.run("I love Paris in the summer."))
# {'embedding': [0.002, 0.032, 0.504, ...]}
AmazonBedrockTextEmbedder.__init__
def __init__(
model: Literal[
"amazon.titan-embed-text-v1",
"cohere.embed-english-v3",
"cohere.embed-multilingual-v3",
"amazon.titan-embed-text-v2:0",
],
aws_access_key_id: Optional[Secret] = Secret.from_env_var(
"AWS_ACCESS_KEY_ID", strict=False),
aws_secret_access_key: Optional[Secret] = Secret.
from_env_var( # noqa: B008
"AWS_SECRET_ACCESS_KEY", strict=False),
aws_session_token: Optional[Secret] = Secret.from_env_var(
"AWS_SESSION_TOKEN", strict=False),
aws_region_name: Optional[Secret] = Secret.from_env_var(
"AWS_DEFAULT_REGION", strict=False),
aws_profile_name: Optional[Secret] = Secret.from_env_var("AWS_PROFILE",
strict=False),
boto3_config: Optional[Dict[str, Any]] = None,
**kwargs)
Initializes the AmazonBedrockTextEmbedder with the provided parameters. The parameters are passed to the
Amazon Bedrock client.
Note that the AWS credentials are not required if the AWS environment is configured correctly. These are loaded
automatically from the environment or the AWS configuration file and do not need to be provided explicitly via
the constructor. If the AWS environment is not configured users need to provide the AWS credentials via the
constructor. Aside from model, three required parameters are aws_access_key_id
, aws_secret_access_key
,
and aws_region_name
.
Arguments:
model
: The embedding model to use. The model has to be specified in the format outlined in the Amazon Bedrock documentation.aws_access_key_id
: AWS access key ID.aws_secret_access_key
: AWS secret access key.aws_session_token
: AWS session token.aws_region_name
: AWS region name.aws_profile_name
: AWS profile name.boto3_config
: The configuration for the boto3 client.kwargs
: Additional parameters to pass for model inference. For example,input_type
andtruncate
for Cohere models.
Raises:
ValueError
: If the model is not supported.AmazonBedrockConfigurationError
: If the AWS environment is not configured correctly.
AmazonBedrockTextEmbedder.run
@component.output_types(embedding=List[float])
def run(text: str)
Embeds the input text using the Amazon Bedrock model.
Arguments:
text
: The input text to embed.
Raises:
TypeError
: If the input text is not a string.AmazonBedrockInferenceError
: If the model inference fails.
Returns:
A dictionary with the following keys:
embedding
: The embedding of the input text.
AmazonBedrockTextEmbedder.to_dict
def to_dict() -> Dict[str, Any]
Serializes the component to a dictionary.
Returns:
Dictionary with serialized data.
AmazonBedrockTextEmbedder.from_dict
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AmazonBedrockTextEmbedder"
Deserializes the component from a dictionary.
Arguments:
data
: Dictionary to deserialize from.
Returns:
Deserialized component.
Module haystack_integrations.components.embedders.amazon_bedrock.document_embedder
AmazonBedrockDocumentEmbedder
A component for computing Document embeddings using Amazon Bedrock.
The embedding of each Document is stored in the embedding
field of the Document.
Usage example:
import os
from haystack.dataclasses import Document
from haystack_integrations.components.embedders.amazon_bedrock import AmazonBedrockDocumentEmbedder
os.environ["AWS_ACCESS_KEY_ID"] = "..."
os.environ["AWS_SECRET_ACCESS_KEY_ID"] = "..."
os.environ["AWS_REGION_NAME"] = "..."
embedder = AmazonBedrockDocumentEmbedder(
model="cohere.embed-english-v3",
input_type="search_document",
)
doc = Document(content="I love Paris in the winter.", meta={"name": "doc1"})
result = embedder.run([doc])
print(result['documents'][0].embedding)
# [0.002, 0.032, 0.504, ...]
AmazonBedrockDocumentEmbedder.__init__
def __init__(
model: Literal[
"amazon.titan-embed-text-v1",
"cohere.embed-english-v3",
"cohere.embed-multilingual-v3",
"amazon.titan-embed-text-v2:0",
],
aws_access_key_id: Optional[Secret] = Secret.from_env_var(
"AWS_ACCESS_KEY_ID", strict=False),
aws_secret_access_key: Optional[Secret] = Secret.
from_env_var( # noqa: B008
"AWS_SECRET_ACCESS_KEY", strict=False),
aws_session_token: Optional[Secret] = Secret.from_env_var(
"AWS_SESSION_TOKEN", strict=False),
aws_region_name: Optional[Secret] = Secret.from_env_var(
"AWS_DEFAULT_REGION", strict=False),
aws_profile_name: Optional[Secret] = Secret.from_env_var("AWS_PROFILE",
strict=False),
batch_size: int = 32,
progress_bar: bool = True,
meta_fields_to_embed: Optional[List[str]] = None,
embedding_separator: str = "\n",
boto3_config: Optional[Dict[str, Any]] = None,
**kwargs)
Initializes the AmazonBedrockDocumentEmbedder with the provided parameters. The parameters are passed to the
Amazon Bedrock client.
Note that the AWS credentials are not required if the AWS environment is configured correctly. These are loaded
automatically from the environment or the AWS configuration file and do not need to be provided explicitly via
the constructor. If the AWS environment is not configured users need to provide the AWS credentials via the
constructor. Aside from model, three required parameters are aws_access_key_id
, aws_secret_access_key
,
and aws_region_name
.
Arguments:
model
: The embedding model to use. The model has to be specified in the format outlined in the Amazon Bedrock documentation.aws_access_key_id
: AWS access key ID.aws_secret_access_key
: AWS secret access key.aws_session_token
: AWS session token.aws_region_name
: AWS region name.aws_profile_name
: AWS profile name.batch_size
: Number of Documents to encode at once. Only Cohere models support batch inference. This parameter is ignored for Amazon Titan models.progress_bar
: Whether to show a progress bar or not. Can be helpful to disable in production deployments to keep the logs clean.meta_fields_to_embed
: List of meta fields that should be embedded along with the Document text.embedding_separator
: Separator used to concatenate the meta fields to the Document text.boto3_config
: The configuration for the boto3 client.kwargs
: Additional parameters to pass for model inference. For example,input_type
andtruncate
for Cohere models.
Raises:
ValueError
: If the model is not supported.AmazonBedrockConfigurationError
: If the AWS environment is not configured correctly.
AmazonBedrockDocumentEmbedder.run
@component.output_types(documents=List[Document])
def run(documents: List[Document])
Embed the provided Document
s using the specified model.
Arguments:
documents
: TheDocument
s to embed.
Raises:
AmazonBedrockInferenceError
: If the inference fails.
Returns:
A dictionary with the following keys:
documents
: TheDocument
s with theembedding
field populated.
AmazonBedrockDocumentEmbedder.to_dict
def to_dict() -> Dict[str, Any]
Serializes the component to a dictionary.
Returns:
Dictionary with serialized data.
AmazonBedrockDocumentEmbedder.from_dict
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AmazonBedrockDocumentEmbedder"
Deserializes the component from a dictionary.
Arguments:
data
: Dictionary to deserialize from.
Returns:
Deserialized component.