With PromptNode, you can use large language models directly or in your pipelines.

What are large language models?

Large language models are huge models trained on enormous amounts of data. Interacting with such a model resembles talking to another person. These models have general knowledge of the world. You can ask them anything, and they'll be able to answer.

Large language models are trained to perform many NLP tasks with little training data. What's astonishing about them is that a single model can perform various NLP tasks with good accuracy.

Some examples of large language models include flan-t5-base, flan-paLM, chinchilla, and GPT-3 variants, such as text-davinci-003.

PromptNode is a very versatile node. It's used in query pipelines, but its position depends on what you want it to do. You can pass a template to specify the NLP task the PromptNode should perform and a model to use. For more information, see the Usage section.


Position in a Pipeline	Used in query pipelines. The position depends on the NLP task you want it to do.
Input	Depends on the NLP task it performs. Some examples are query, documents, output of the preceding node.
Output	Depends on the NLP task it performs. Some examples are answer, query, document summary.
Classes	PromptNode

Usage

You can use PromptNode as a stand-alone node or in a pipeline. If you don't specify the model you want to use for the node, it uses flan t5 base.

Stand Alone

Just initialize the node and ask a question. The model has general knowledge about the world, so you can ask it anything:

from haystack.nodes import PromptNode

# Initialize the node:
prompt_node = PromptNode()

# Run a prompt
prompt_node("What is the capital of Germany?")

# Here's the output:
['berlin']

With a Template

For better results, use a task-specific template to create your prompt. You can pass additional variables like documents or questions to the node. The template combines all inputs into a single prompt:

from haystack.nodes import PromptNode, PromptTemplate

# Initalize the node
prompt_node = PromptNode()

# Specify the template using the `prompt` method 
# and pass your documents and questions:
prompt_node.prompt(prompt_template="question-answering", 
          documents=["Berlin is the capital of Germany.", "Paris is the capital of France."],
          questions=["What is the capital of Germany?", "What is the capital of France"])

# Here's the output:
['Berlin', 'Paris']

To explore the real power of templates, see the Templates section.

With a Model Specified

By default, the PromptNode uses the flan t5 base model. You can also use other google/flan t_5 models and the davinci model by OpenAI.

from haystack.nodes import PromptNode

# Initalize the node passing the model:
prompt_node = PromptNode(model_name_or_path="google/flan-t5-xl")

# Go ahead and ask a question:
prompt_node("What is the best city in Europe to live in?")

In a Pipeline

The real power of PromptNode shows when you use it in a pipeline. Look at the example to get an idea of what's possible.

Long-Form Question Answering

Long-form QA is one use of the PromptNode, but certainly not the only one. In this QA type, PromptNode handles complex questions by synthesizing information from various documents to retrieve an answer.

from haystack.pipelines import Pipeline
from haystack.nodes import Shaper, PromptNode, PromptTemplate
from haystack.schema import Document

# Let's create a custom LFQA prompt using PromptTemplate
lfqa_prompt = PromptTemplate(name="lfqa",
                             prompt_text="""Synthesize a comprehensive answer from the following topk most relevant paragraphs and the given question. 
                             Provide a clear and concise response that summarizes the key points and information presented in the paragraphs. 
                             Your answer should be in your own words and be no longer than 50 words. 
                             \n\n Paragraphs: $documents \n\n Question: $query \n\n Answer:""") 

# These docs could also come from a retriever
# Here we explicitly specify them to avoid the setup steps for Retriever and DocumentStore
doc_1 = "Contrails are a manmade type of cirrus cloud formed when water vapor from the exhaust of a jet engine condenses on particles, which come from either the surrounding air or the exhaust itself, and freezes, leaving behind a visible trail. The exhaust can also trigger the formation of cirrus by providing ice nuclei when there is an insufficient naturally-occurring supply in the atmosphere. One of the environmental impacts of aviation is that persistent contrails can form into large mats of cirrus, and increased air traffic has been implicated as one possible cause of the increasing frequency and amount of cirrus in Earth's atmosphere."
doc_2 = "Because the aviation industry is especially sensitive to the weather, accurate weather forecasting is essential. Fog or exceptionally low ceilings can prevent many aircraft from landing and taking off. Turbulence and icing are also significant in-flight hazards. Thunderstorms are a problem for all aircraft because of severe turbulence due to their updrafts and outflow boundaries, icing due to the heavy precipitation, as well as large hail, strong winds, and lightning, all of which can cause severe damage to an aircraft in flight. Volcanic ash is also a significant problem for aviation, as aircraft can lose engine power within ash clouds. On a day-to-day basis airliners are routed to take advantage of the jet stream tailwind to improve fuel efficiency. Aircrews are briefed prior to takeoff on the conditions to expect en route and at their destination. Additionally, airports often change which runway is being used to take advantage of a headwind. This reduces the distance required for takeoff, and eliminates potential crosswinds."

# Shaper concatenates the most relevant docs into one doc used as context for the generated answer
shaper = Shaper(func="join_documents", inputs={"documents": "documents"}, outputs=["documents"])

# Let's initiate the PromptNode 
node = PromptNode("text-davinci-003", default_prompt_template=lfqa_prompt, api_key=api_key)

# Let's create a pipeline with Shaper and PromptNode
pipe = Pipeline()
pipe.add_node(component=shaper, name="shaper", inputs=["Query"])
pipe.add_node(component=node, name="prompt_node", inputs=["shaper"])

output = pipe.run(query="Why do airplanes leave contrails in the sky?", documents=[Document(doc_1), Document(doc_2)])
output["results"]

# Here's the answer:
["Contrails are manmade clouds formed when water vapor from the exhaust of a jet engine condenses on particles, which come from either the surrounding air or the exhaust itself, and freezes, creating a visible trail. Increased air traffic has been linked to the greater frequency and amount of these cirrus clouds in Earth's atmosphere."]

To learn more about the template structure, see the API documentation and the Templates section below.

PromptTemplates

PromptNode comes with out-of-the-box prompt templates ready for you to use. A template corresponds to an NLP task. Here are some of the templates you can specify for the PromptNode:

question-answering
question-generation
summarization
conditioned-question-generation
question-answering-check
sentiment-analysis
topic-classification
multiple-choice-question-answering
language-detection
translation

For a full list of ready-to-use templates, see PromptNode in the Github repo.

If you don't specify the template, the node tries to guess what task you want it to perform. By indicating the template, you ensure it performs the right task.

PromptTemplate Structure

Here's an example of the question-answering template:

PromptTemplate(
        name="question-answering",
        prompt_text="Given the context please answer the question. Context: $documents; Question: $questions; Answer:"
    )

The prompt_text parameter contains the prompt template text for the task you want the model to do. It also specifies input variables: documents and questions. At runtime, these variables must be present in the execution context of the node. For a detailed structure of each template, see the API documentation.

Adding Your Prompt

You can easily add your template:

from haystack.nodes import PromptTemplate, PromptNode

# In `prompt_text`, tell the model what you want it to do.
PromptNode.add_prompt_template(PromptTemplate(name="sentiment-analysis-new",
                                              prompt_text="Indicate the sentiment. Answer with positive, negative, or neutral. Context: $documents; Answer:"))

For guidelines on how to construct most efficient prompts, see Prompt Engineering Guidelines.

Setting a Default PromptTemplate

You can set a default template for a PromptNode instance. This way, you can reuse the same PromptNode in your pipeline for different tasks:

from haystack.nodes import PromptTemplate, PromptNode
from haystack.schema import Document

prompt_node = PromptNode()
sa = prompt_node.set_default_prompt_template("sentiment-analysis-new")
sa(documents=[Document("I am in love and I feel great!")])

# Node output:
['positive']

# You can then switch to another template:
summarizer = sa.set_default_prompt_template("summarization")

Models

The default model for PromptModel and PromptNode is google/flan-t5-base but you can use any other LLM. To do this, specify the model's name and the API key.

Using Another Model

You can replace the default model with a flan t5 model of a different size or a model by OpenAI.
This example uses a version of the GPT-3 model:

from haystack.nodes import PromptModel, PromptNode

openai_api_key = <type your OpenAI API key>

# Specify the model you want to use:
prompt_open_ai = PromptModel(model_name_or_path="text-davinci-003", api_key=openai_api_key)

# Make PromptNode use the model:
pn_open_ai = PromptNode(prompt_open_ai)

pn_open_ai("What's the coolest city to live in Germany?")

Using Different Models in One Pipeline

You can also specify different LLMs for each PromptNode in your pipeline.

from haystack.nodes. import PromptTemplate, PromptNode, PromptModel
from haystack.pipelines import Pipeline

# This is to set up the OpenAI model:
from getpass import getpass

api_key_prompt = "Enter OpenAI API key:" 
api_key = getpass(api_key_prompt)

# Specify the model you want to use:
prompt_open_ai = PromptModel(model_name_or_path="text-davinci-003", api_key=api_key)

# This sets up the default model:
prompt_model = PromptModel()

# Now let make one PromptNode use the default model and the other one the OpenAI model:
node_default_model = PromptNode(prompt_model, default_prompt_template="question-generation", output_variable="questions")
node_openai = PromptNode(prompt_open_ai, default_prompt_template="question-answering")

pipeline = Pipeline()
pipeline.add_node(component=node_default_model, name="prompt_node1", inputs=["Query"])
pipe.add_node(component=node_openai, name="prompt_node_2", inputs=["prompt_node1"])
output = pipe.run(query="not relevant", documents=[Document("Berlin is the capital of Germany")])
output["results"]