HomeDocumentationAPI ReferenceTutorials
Haystack Homepage


PromptNode brings you the power of large language models. It's an easy-to-use, customizable node that you can run on its own or in your pipelines for various NLP tasks.

With PromptNode, you can use large language models directly or in your pipelines.

What are large language models?

Large language models are huge models trained on enormous amounts of data. Interacting with such a model resembles talking to another person. These models have general knowledge of the world. You can ask them anything, and they'll be able to answer.

Large language models are trained to perform many NLP tasks with little training data. What's astonishing about them is that a single model can perform various NLP tasks with good accuracy.

Some examples of large language models include flan-t5-base, flan-paLM, chinchilla, and GPT-3 variants, such as text-davinci-003.

PromptNode is a very versatile node. It's used in query pipelines, but its position depends on what you want it to do. You can pass a template to specify the NLP task the PromptNode should perform and a model to use. For more information, see the Usage section.

Position in a PipelineUsed in query pipelines. The position depends on the NLP task you want it to do.
InputDepends on the NLP task it performs. Some examples are query, documents, output of the preceding node.
OutputDepends on the NLP task it performs. Some examples are answer, query, document summary.


You can use PromptNode as a stand-alone node or in a pipeline. If you don't specify the model you want to use for the node, it uses flan t5 base.

Stand Alone

Just initialize the node and ask a question. The model has general knowledge about the world, so you can ask it anything:

from haystack.nodes.prompt import PromptNode

# Initialize the node:
prompt_node = PromptNode()

# Run a prompt
prompt_node("What is the capital of Germany?")

# Here's the output:

With a Template

For better results, use a task-specific template to create your prompt. You can pass additional variables like documents or questions to the node. The template combines all inputs into a single prompt:

from haystack.nodes.prompt import PromptNode

# Initalize the node
prompt_node = PromptNode()

# Specify the template using the `prompt` method 
# and pass your documents and questions:
          documents=["Berlin is the capital of Germany.", "Paris is the capital of France."],
          questions=["What is the capital of Germany?", "What is the capital of France"])

# Here's the output:
['Berlin', 'Paris']

To explore the real power of templates, see the Pipelines with Multiple PromptNodes and the Templates sections.

With a Model Specified

By default, the PromptNode uses the flan t5 base model. You can also use other google/flan t_5 models and the davinci model by OpenAI.

from haystack.nodes.prompt import PromptNode

# Initalize the node passing the model:
prompt_node = PromptNode(model_name_or_path="google/flan-t5-xl")

# Go ahead and ask a question:
prompt_node("What is the best city in Europe to live in?")

In a Pipeline

The real power of PromptNode shows when you use it in a pipeline. Look at the examples to get an idea of what's possible.

Question Generation on Retrieved Documents

In this example, the Retriever retrieves the documents from the DocumentStore. Then, the PromptNode uses the retrieved documents to generate questions:

from haystack.nodes.prompt import PromptNode, PromptTemplate
from haystack.pipelines import Pipeline

top_k = 3
query = "Who are the parents of Arya Stark?"

# Set the template to the question-generation template
node = PromptNode(default_prompt_template="question-generation")
pipeline = Pipeline()
# First, we're adding a Retriever to retrieve only a set of documents, based on the query
pipeline.add_node(component=retriever, name="retriever", inputs=["Query"])
# Here, we're injecting the retrieved documents to the PromptNode template.
# The PromptNote will use these documents to generate questions.
pipeline.add_node(component=node, name="prompt_node", inputs=["retriever"])

output = pipeline.run(query=query)


Pipelines with Multiple PromptNodes

You can use many PromptNodes in one pipeline, each performing a different task. Just set a default prompt template for each PromptNode.

If you're using multiple PromptNodes, we recommend you reuse the instance of the model. LLMs are huge and take a lot of memory. If you use multiple PromptNodes and each of them uses a model that's not an online one, you can run out of memory.

When you add the PromptNode instances to the pipeline, the output of one instance is used as input by another instance.

Here's an example of a pipeline where the first instance of the PromptNode generates questions, then the second instance answers them:

from haystack.nodes.prompt import PromptNode, PromptTemplate
from haystack.pipelines import Pipeline
from haystack.nodes import EmbeddingRetriever

# First, let's initiate the Retriever:
retriever = EmbeddingRetriever(

# Now, let's set the number of documents we want to retrieve:
top_k = 10

# And the Query we want the Retriever to use for retrieving the documents:
query = "Who are the parents of Arya Stark?"

# Now, we're defining the shared model for each PromptNode:
prompt_model = PromptModel()

# Let's initialize the PromptNodes. Notice that they share the prompt model.
# This initalizes the PromptNode that's going to generate the questions:
qg_prompt_node = PromptNode(prompt_model, default_prompt_template="question-generation", output_variable="questions")

# And this initializes the PromptNode that's going to answer the questions:
qa_prompt_node = PromptNode(prompt_model, default_prompt_template="question-answering")

# Time to define the pipeline:
pipeline = Pipeline()
pipeline.add_node(component=retriever, name="retriever", inputs=["Query"])
pipeline.add_node(component=qg_prompt_node, name="qg_prompt_node", inputs=["retriever"])
pipeline.add_node(component=qa_prompt_node, name="qa_prompt_node", inputs=["qg_prompt_node"])
output = pipeline.run(query=query)

You can do the same thing without a Retriever:

from haystack.schema import Document
from haystack.nodes.prompt import PromptNode, PromptModel, PromptTemplate
from haystack.pipelines import Pipeline

# Define the model to reuse for each PromptNode:
prompt_model = PromptModel()

# Define two PromptNodes, one to generate the questions, the other to answer the generated questions:
qg_prompt_node = PromptNode(prompt_model, default_prompt_template="question-generation", output_variable="questions")
qa_prompt_node = PromptNode(prompt_model, default_prompt_template="question-answering")

pipeline = Pipeline()
pipeline.add_node(component=qg_prompt_node, name="prompt_node_1", inputs=["Query"])
pipeline.add_node(component=qa_prompt_node, name="prompt_node_2", inputs=["prompt_node_1"])
output = pipeline.run(query="not relevant", documents=[Document("Berlin is the capital of Germany")])

The output_variable of the PromptNode generating the questions is questions because we want to use the generated questions in the question answering template of the downstream PromptNode. The question answering template defines questions as the input variable.

To learn more about the template structure, see the API documentation and the Templates section below.


PromptNode comes with out-of-the-box templates ready for you to use. A template corresponds to an NLP task. Here are some of the templates you can specify for the PromptNode:

  • question-answering
  • question-generation
  • summarization
  • conditioned-question-generation
  • question-answering-check
  • sentiment-analysis
  • topic-classification
  • multiple-choice-question-answering
  • language-detection
  • translation

If you don't specify the template, the node tries to guess what task you want it to perform. By indicating the template, you ensure it performs the right task.

Template Structure

Here's an example of the question-answering template:

        prompt_text="Given the context please answer the question. Context: $documents; Question: $questions; Answer:"

The prompt_text parameter contains the prompt template text for the task you want the model to do. It also specifies input variables: documents and questions. At runtime, these variables must be present in the execution context of the node. For a detailed structure of each template, see the API documentation.

Adding Your Template

You can easily add your template:

from haystack.nodes.prompt import PromptTemplate, PromptNode

# In `prompt_text`, tell the model what you want it to do.
                                              prompt_text="Indicate the sentiment. Answer with positive, negative, or neutral. Context: $documents; Answer:"))

Setting a Default Template

You can set a default template for a PromptNode instance. This way, you can reuse the same PromptNode in your pipeline for different tasks:

from haystack.nodes.prompt import PromptTemplate, PromptNode

prompt_node = PromptNode()
sa = prompt_node.set_default_prompt_template("sentiment-analysis-new")
sa(documents=[Document("I am in love and I feel great!")])

# Node output:

# You can then switch to another template:
summarizer = sa.set_default_prompt_template("summarization")


The default model for PromptModel and PromptNode is google/flan-t5-base but you can use any other LLM model. To do this, specify the model's name and the API key.

Using Another Model

You can replace the default model with a flan t5 model of a different size or a model by OpenAI.
This example uses a version of the GPT-3 model:

from haystack.nodes.prompt import PromptModel, PromptNode

openai_api_key = <type your OpenAI API key>

# Specify the model you want to use:
prompt_open_ai = PromptModel(model_name_or_path="text-davinci-003", api_key=openai_api_key)

# Make PromptNode use the model:
pn_open_ai = PromptNode(prompt_open_ai)

pn_open_ai("What's the coolest city to live in Germany?")

Using Different Models in One Pipeline

You can also specify different LLMs for each PromptNode in your pipeline.

from haystack.nodes.prompt import PromptTemplate, PromptNode, PromptModel

# This is to set up the OpenAI model:
from getpass import getpass

api_key_prompt = "Enter OpenAI API key:" 
api_key = getpass(api_key_prompt)

# Specify the model you want to use:
prompt_open_ai = PromptModel(model_name_or_path="text-davinci-003", api_key=api_key)

# This sets up the default model:
prompt_model = PromptModel()

# Now let make one PromptNode use the default model and the other one the OpenAI model:
node_default_model = PromptNode(prompt_model, default_prompt_template="question-generation", output_variable="questions")
node_openai = PromptNode(prompt_open_ai, default_prompt_template="question-answering")

pipeline = Pipeline()
pipeline.add_node(component=node_default_model, name="prompt_node1", inputs=["Query"])
pipe.add_node(component=node_openai, name="prompt_node_2", inputs=["prompt_node1"])
output = pipe.run(query="not relevant", documents=[Document("Berlin is the capital of Germany")])

Related Links