Module prompt_node

PromptNode

class PromptNode(BaseComponent)

The PromptNode class is the central abstraction in Haystack's large language model (LLM) support. PromptNode supports multiple NLP tasks out of the box. You can use it to perform tasks such as summarization, question answering, question generation, and more, using a single, unified model within the Haystack framework.

One of the benefits of PromptNode is that you can use it to define and add additional prompt templates the model supports. Defining additional prompt templates makes it possible to extend the model's capabilities and use it for a broader range of NLP tasks in Haystack. Prompt engineers define templates for each NLP task and register them with PromptNode. The burden of defining templates for each task rests on the prompt engineers, not the users.

Using an instance of the PromptModel class, you can create multiple PromptNodes that share the same model, saving the memory and time required to load the model multiple times.

PromptNode also supports multiple model invocation layers:

• Hugging Face transformers (all text2text-generation models)
• OpenAI InstructGPT models
• Azure OpenAI InstructGPT models

But you're not limited to the models listed above, as you can register additional custom model invocation layers.

We recommend using LLMs fine-tuned on a collection of datasets phrased as instructions, otherwise we find that the LLM does not "follow" prompt instructions well. The list of instruction-following models increases every month, and the current list includes: Flan, OpenAI InstructGPT, opt-iml, bloomz, and mt0 models.

For more details, see PromptNode.

PromptNode.__init__

def __init__(model_name_or_path: Union[str,
                                       PromptModel] = "google/flan-t5-base",
             default_prompt_template: Optional[Union[str,
                                                     PromptTemplate]] = None,
             output_variable: Optional[str] = None,
             max_length: Optional[int] = 100,
             api_key: Optional[str] = None,
             api_base: Optional[str] = None,
             timeout: Optional[float] = None,
             use_auth_token: Optional[Union[str, bool]] = None,
             use_gpu: Optional[bool] = None,
             devices: Optional[List[Union[str, "torch.device"]]] = None,
             stop_words: Optional[List[str]] = None,
             top_k: int = 1,
             debug: Optional[bool] = False,
             model_kwargs: Optional[Dict] = None,
             truncate: bool = True)

Creates a PromptNode instance.

Arguments:

• model_name_or_path: The name of the model to use or an instance of the PromptModel.
• default_prompt_template: The default prompt template to use for the model.
• output_variable: The name of the output variable in which you want to store the inference results. If not set, PromptNode uses PromptTemplate's output_variable. If PromptTemplate's output_variable is not set, the default name is results.
• max_length: The maximum number of tokens the generated text output can have.
• api_key: The API key to use for the model.
• use_auth_token: The authentication token to use for the model.
• use_gpu: Whether to use GPU or not.
• devices: The devices to use for the model.
• top_k: The number of independently generated texts to return per prompt. For example, if you set top_k=3, the model will generate three answers to the query.
• stop_words: Stops text generation if any of the stop words is generated.
• model_kwargs: Additional keyword arguments passed when loading the model specified in model_name_or_path.
• debug: Whether to include the used prompts as debug information in the output under the key _debug.
• truncate: Whether to truncate the prompt to the maximum token limit before sending it to the model. Note that Azure OpenAI InstructGPT models require two additional parameters: azure_base_url (the URL for the Azure OpenAI API endpoint, usually in the form https://<your-endpoint>.openai.azure.com') and azure_deployment_name (the name of the Azure OpenAI API deployment). You should specify these parameters in the model_kwargs` dictionary.

PromptNode.__call__

def __call__(*args, **kwargs) -> List[Any]

This method is invoked when the component is called directly, for example:

    PromptNode pn = ...
    sa = pn.set_default_prompt_template("sentiment-analysis")
    sa(documents=[Document("I am in love and I feel great!")])

PromptNode.prompt

def prompt(prompt_template: Optional[Union[str, PromptTemplate]], *args,
           **kwargs) -> List[Any]

Prompts the model and represents the central API for the PromptNode. It takes a prompt template,

a list of non-keyword and keyword arguments, and returns a list of strings - the responses from the underlying model.

If you specify the optional prompt_template parameter, it takes precedence over the default PromptTemplate for this PromptNode.

Arguments:

• prompt_template: The name or object of the optional PromptTemplate to use.

Returns:

A list of strings as model responses.

PromptNode.default_prompt_template

@default_prompt_template.setter
def default_prompt_template(prompt_template: Union[str, PromptTemplate, None])

Sets the default prompt template for the node.

Arguments:

• prompt_template: The prompt template to be set as default.

Returns:

The current PromptNode object.

PromptNode.get_prompt_template

def get_prompt_template(
    prompt_template: Union[str, PromptTemplate, None] = None
) -> Optional[PromptTemplate]

Resolves a prompt template.

Arguments:

• prompt_template: The prompt template to be resolved. You can choose between the following types:
• None: Returns the default prompt template.
• PromptTemplate: Returns the given prompt template object.
• str: Parses the string depending on its content:
  • prompt template name: Returns the prompt template registered with the given name.
  • prompt template yaml: Returns a prompt template specified by the given YAML.
  • prompt text: Returns a copy of the default prompt template with the given prompt text.

:return: The prompt template object.

PromptNode.prompt_template_params

def prompt_template_params(prompt_template: str) -> List[str]

Returns the list of parameters for a prompt template.

Arguments:

• prompt_template: The name of the prompt template.

Returns:

The list of parameters for the prompt template.

PromptNode.run

def run(query: Optional[str] = None,
        file_paths: Optional[List[str]] = None,
        labels: Optional[MultiLabel] = None,
        documents: Optional[List[Document]] = None,
        meta: Optional[dict] = None,
        invocation_context: Optional[Dict[str, Any]] = None,
        prompt_template: Optional[Union[str, PromptTemplate]] = None,
        generation_kwargs: Optional[Dict[str,
                                         Any]] = None) -> Tuple[Dict, str]

Runs the PromptNode on these input parameters. Returns the output of the prompt model.

The parameters query, file_paths, labels, documents, and meta are added to the invocation context before invoking the prompt model. PromptNode uses these variables only if they are present as parameters in the PromptTemplate.

Arguments:

• query: The PromptNode usually ignores the query, unless it's used as a parameter in the prompt template.
• file_paths: The PromptNode usually ignores the file paths, unless they're used as a parameter in the prompt template.
• labels: The PromptNode usually ignores the labels, unless they're used as a parameter in the prompt template.
• documents: The documents to be used for the prompt.
• meta: PromptNode usually ignores meta information, unless it's used as a parameter in the PromptTemplate.
• invocation_context: The invocation context to be used for the prompt.
• prompt_template: The prompt template to use. You can choose between the following types:
• None: Use the default prompt template.
• PromptTemplate: Use the given prompt template object.
• str: Parses the string depending on its content:
  • prompt template name: Uses the prompt template registered with the given name.
  • prompt template yaml: Uses the prompt template specified by the given YAML.
  • prompt text: Uses a copy of the default prompt template with the given prompt text.
• generation_kwargs: The generation_kwargs are used to customize text generation for the underlying pipeline.

PromptNode.arun

async def arun(
        query: Optional[str] = None,
        file_paths: Optional[List[str]] = None,
        labels: Optional[MultiLabel] = None,
        documents: Optional[List[Document]] = None,
        meta: Optional[dict] = None,
        invocation_context: Optional[Dict[str, Any]] = None,
        prompt_template: Optional[Union[str, PromptTemplate]] = None,
        generation_kwargs: Optional[Dict[str,
                                         Any]] = None) -> Tuple[Dict, str]

Drop-in replacement asyncio version of the run method, see there for documentation.

PromptNode.run_batch

def run_batch(queries: Optional[List[str]] = None,
              documents: Optional[Union[List[Document],
                                        List[List[Document]]]] = None,
              invocation_contexts: Optional[List[Dict[str, Any]]] = None,
              prompt_templates: Optional[List[Union[str,
                                                    PromptTemplate]]] = None)

Runs PromptNode in batch mode.

• If you provide a list containing a single query (or invocation context)...

  • ... and a single list of Documents, the query is applied to each Document individually.
  • ... and a list of lists of Documents, the query is applied to each list of Documents and the results are aggregated per Document list.

• If you provide a list of multiple queries (or multiple invocation contexts)...

  • ... and a single list of Documents, each query (or invocation context) is applied to each Document individually.
  • ... and a list of lists of Documents, each query (or invocation context) is applied to its corresponding list of Documents and the results are aggregated per query-Document pair.

• If you provide no Documents, then each query (or invocation context) is applied directly to the PromptTemplate.

Arguments:

• queries: List of queries.
• documents: Single list of Documents or list of lists of Documents in which to search for the answers.
• invocation_contexts: List of invocation contexts.
• prompt_templates: The prompt templates to use. You can choose between the following types:
• None: Use the default prompt template.
• PromptTemplate: Use the given prompt template object.
• str: Parses the string depending on its content:
  • prompt template name: Uses the prompt template registered with the given name.
  • prompt template yaml: Uuses the prompt template specified by the given YAML.
  • prompt text: Uses a copy of the default prompt template with the given prompt text.

Module prompt_model

PromptModel

class PromptModel(BaseComponent)

The PromptModel class is a component that uses a pre-trained model to perform tasks defined in a prompt. Out of the box, it supports model invocation layers for:

• Hugging Face transformers (all text2text-generation and text-generation models)
• OpenAI InstructGPT models
• Azure OpenAI InstructGPT models

Although it's possible to use PromptModel to make prompt invocations on the underlying model, use PromptNode to interact with the model. PromptModel instances are a way for multiple PromptNode instances to use a single PromptNode, and thus save computational resources.

For more details, refer to PromptModels.

PromptModel.__init__

def __init__(model_name_or_path: str = "google/flan-t5-base",
             max_length: Optional[int] = 100,
             api_key: Optional[str] = None,
             api_base: Optional[str] = None,
             timeout: Optional[float] = None,
             use_auth_token: Optional[Union[str, bool]] = None,
             use_gpu: Optional[bool] = None,
             devices: Optional[List[Union[str, "torch.device"]]] = None,
             invocation_layer_class: Optional[Union[
                 Type[PromptModelInvocationLayer], str]] = None,
             model_kwargs: Optional[Dict] = None)

Creates an instance of PromptModel.

Arguments:

• model_name_or_path: The name or path of the underlying model.
• max_length: The maximum number of tokens the output text generated by the model can have.
• api_key: The API key to use for the model.
• use_auth_token: The Hugging Face token to use.
• use_gpu: Whether to use GPU or not.
• devices: The devices to use where the model is loaded.
• invocation_layer_class: The custom invocation layer class to use. If None, known invocation layers are used.
• model_kwargs: Additional keyword arguments passed to the underlying model. Note that Azure OpenAI InstructGPT models require two additional parameters: azure_base_url (The URL for the Azure OpenAI API endpoint, usually in the form https://<your-endpoint>.openai.azure.com') and azure_deployment_name (the name of the Azure OpenAI API deployment). You should add these parameters in the model_kwargs` dictionary.

PromptModel.invoke

def invoke(prompt: Union[str, List[str], List[Dict[str, str]]],
           **kwargs) -> List[str]

Takes in a prompt and returns a list of responses using the underlying invocation layer.

Arguments:

• prompt: The prompt to use for the invocation. It can be a single prompt or a list of prompts.
• kwargs: Additional keyword arguments to pass to the invocation layer.

Returns:

A list of model-generated responses for the prompt or prompts.

PromptModel.ainvoke

async def ainvoke(prompt: Union[str, List[str], List[Dict[str, str]]],
                  **kwargs) -> List[str]

Drop-in replacement asyncio version of the invoke method, see there for documentation.

Module prompt_template

fetch_from_prompthub

@tenacity.retry(
    reraise=True,
    retry=tenacity.retry_if_exception_type(
        (HTTPError, RequestException, JSONDecodeError)),
    wait=tenacity.wait_exponential(multiplier=PROMPTHUB_BACKOFF),
    stop=tenacity.stop_after_attempt(PROMPTHUB_MAX_RETRIES),
)
def fetch_from_prompthub(name: str) -> prompthub.Prompt

Looks for the given prompt in the PromptHub.

Arguments:

• name: the name of the prompt on the Hub.

Returns:

the Prompt object.

cache_prompt

def cache_prompt(data: prompthub.Prompt)

Saves the prompt to the cache. Helps avoiding naming mismatches in the cache folder.

Arguments:

• data: the prompthub.Prompt object from PromptHub.

PromptTemplate

class PromptTemplate(BasePromptTemplate, ABC)

PromptTemplate is a template for the prompt you feed to the model to instruct it what to do. For example, if you want the model to perform sentiment analysis, you simply tell it to do that in a prompt. Here's what a prompt template may look like:

```python
    PromptTemplate("Give a sentiment for this context. Answer with positive, negative or neutral. Context: {documents}; Answer:")
```

Optionally, you can declare prompt parameters using f-string syntax in the PromptTemplate. Prompt parameters are input parameters that need to be filled in
the prompt_text for the model to perform the task. For example, in the template above, there's one prompt parameter, `documents`.

You declare prompt parameters by adding variables to the prompt text. These variables should be in the format: `{variable}`. In the template above, the variable is `{documents}`.

At runtime, the variables you declared in prompt text are filled in with the arguments passed to the `fill()` method of the PromptTemplate. So in the example above, the `{documents}` variable will be filled with the Documents whose sentiment you want the model to analyze.

Note that other than strict f-string syntax, you can safely use the following backslash characters in the text parts of the prompt text: `

, , . In f-string expressions, use new_line, tab, carriage_return instead. Double quotes (") are automatically replaced with single quotes (') in the prompt text. If you want to use double quotes in the prompt text, use {double_quote}` instead.

For more details on how to use PromptTemplate, see
[PromptTemplates](https://docs.haystack.deepset.ai/docs/prompt_node#prompttemplates).

PromptTemplate.__init__

def __init__(prompt: str,
             output_parser: Optional[Union[BaseOutputParser,
                                           Dict[str, Any]]] = None)

Creates a PromptTemplate instance.

Arguments:

• prompt: The name of the prompt template on the PromptHub (for example, "sentiment-analysis", "question-generation"), a Path to a local file, or the text of a new prompt, including its parameters.
• output_parser: A parser that applied to the model output. For example, to convert the model output to an Answer object, you can use AnswerParser. Instead of BaseOutputParser instances, you can also pass dictionaries defining the output parsers. For example:

output_parser={"type": "AnswerParser", "params": {"pattern": "Answer: (.*)"}},

PromptTemplate.prepare

def prepare(*args, **kwargs) -> Dict[str, Any]

Prepares and verifies the PromtpTemplate with input parameters.

Arguments:

• args: Non-keyword arguments to fill the parameters in the prompt text of a PromptTemplate.
• kwargs: Keyword arguments to fill the parameters in the prompt text of a PromptTemplate.

Returns:

A dictionary with the prompt text and the prompt parameters.

PromptTemplate.post_process

def post_process(prompt_output: List[str], **kwargs) -> List[Any]

Post-processes the output of the PromptTemplate.

Arguments:

• args: Non-keyword arguments to use for post-processing the prompt output.
• kwargs: Keyword arguments to use for post-processing the prompt output.

Returns:

A dictionary with the post-processed output.

PromptTemplate.fill

def fill(*args, **kwargs) -> Iterator[str]

Fills the parameters defined in the prompt text with the arguments passed to it and returns the iterator prompt text.

You can pass non-keyword (args) or keyword (kwargs) arguments to this method. If you pass non-keyword arguments, their order must match the left-to-right order of appearance of the parameters in the prompt text. For example, if the prompt text is: Come up with a question for the given context and the answer. Context: {documents}; Answer: {answers}; Question:, then the first non-keyword argument fills the {documents} variable and the second non-keyword argument fills the {answers} variable.

If you pass keyword arguments, the order of the arguments doesn't matter. Variables in the prompt text are filled with the corresponding keyword argument.

Arguments:

• args: Non-keyword arguments to fill the parameters in the prompt text. Their order must match the order of appearance of the parameters in the prompt text.
• kwargs: Keyword arguments to fill the parameters in the prompt text.

Returns:

An iterator of prompt texts.

PromptTemplate.remove_template_params

def remove_template_params(kwargs: Dict[str, Any]) -> Dict[str, Any]

Removes template parameters from kwargs.

Arguments:

• kwargs: Keyword arguments to remove template parameters from.

Returns:

A modified dictionary with the template parameters removed.

Module shapers

BaseOutputParser

class BaseOutputParser(Shaper)

An output parser in PromptTemplate defines how to parse the model output and convert it into Haystack primitives (answers, documents, or labels). BaseOutputParser is the base class for output parser implementations.

BaseOutputParser.__init__

def __init__(func: str,
             outputs: List[str],
             inputs: Optional[Dict[str, Union[List[str], str]]] = None,
             params: Optional[Dict[str, Any]] = None,
             publish_outputs: Union[bool, List[str]] = True)

Initializes the Shaper component.

Some examples:

- name: shaper
  type: Shaper
  params:
  func: value_to_list
  inputs:
    value: query
    target_list: documents
  outputs:
    - questions

This node takes the content of query and creates a list that contains the value of query len(documents) times. This list is stored in the invocation context under the key questions.

- name: shaper
  type: Shaper
  params:
  func: join_documents
  inputs:
    value: documents
  params:
    delimiter: ' - '
  outputs:
    - documents

This node overwrites the content of documents in the invocation context with a list containing a single Document whose content is the concatenation of all the original Documents. So if documents contained [Document("A"), Document("B"), Document("C")], this shaper overwrites it with [Document("A - B - C")]

- name: shaper
  type: Shaper
  params:
  func: join_strings
  params:
    strings: ['a', 'b', 'c']
    delimiter: ' . '
  outputs:
    - single_string

- name: shaper
  type: Shaper
  params:
  func: strings_to_documents
  inputs:
    strings: single_string
    metadata:
      name: 'my_file.txt'
  outputs:
    - single_document

These two nodes, executed one after the other, first add a key in the invocation context called single_string that contains a . b . c, and then create another key called single_document that contains instead [Document(content="a . b . c", metadata={'name': 'my_file.txt'})].

Arguments:

• func: The function to apply.
• inputs: Maps the function's input kwargs to the key-value pairs in the invocation context. For example, value_to_list expects the value and target_list parameters, so inputs might contain: {'value': 'query', 'target_list': 'documents'}. It doesn't need to contain all keyword args, see params.
• params: Maps the function's input kwargs to some fixed values. For example, value_to_list expects value and target_list parameters, so params might contain {'value': 'A', 'target_list': [1, 1, 1, 1]} and the node's output is ["A", "A", "A", "A"]. It doesn't need to contain all keyword args, see inputs. You can use params to provide fallback values for arguments of run that you're not sure exist. So if you need query to exist, you can provide a fallback value in the params, which will be used only if query is not passed to this node by the pipeline.
• outputs: The key to store the outputs in the invocation context. The length of the outputs must match the number of outputs produced by the function invoked.
• publish_outputs: Controls whether to publish the outputs to the pipeline's output. Set True (default value) to publishes all outputs or False to publish None. E.g. if outputs = ["documents"] result for publish_outputs = True looks like

    {
        "invocation_context": {
            "documents": [...]
        },
        "documents": [...]
    }

For publish_outputs = False result looks like

    {
        "invocation_context": {
            "documents": [...]
        },
    }

If you want to have finer-grained control, pass a list of the outputs you want to publish.

AnswerParser

class AnswerParser(BaseOutputParser)

Parses the model output to extract the answer into a proper Answer object using regex patterns. AnswerParser adds the document_ids of the documents used to generate the answer and the prompts used to the Answer object. You can pass a reference_pattern to extract the document_ids of the answer from the model output.

AnswerParser.__init__

def __init__(pattern: Optional[str] = None,
             reference_pattern: Optional[str] = None)

Arguments:

• pattern: The regex pattern to use for parsing the answer. Examples: [^\n]+$ finds "this is an answer" in string "this is an argument. this is an answer". Answer: (.*) finds "this is an answer" in string "this is an argument. Answer: this is an answer". If not specified, the whole string is used as the answer. If specified, the first group of the regex is used as the answer. If there is no group, the whole match is used as the answer. :param reference_pattern: The regex pattern to use for parsing the document references. Example: \[(\d+)\] finds "1" in string "this is an answer[1]". If None, no parsing is done and all documents are referenced.