SentenceTransformersDiversityRanker

This is a Diversity Ranker based on Sentence Transformers.

Most common position in a pipeline	In a query pipeline, after a component that returns a list of documents such as a Retriever
Mandatory init variables	None
Mandatory run variables	documents: A list of documents query: A query string
Output variables	documents: A list of documents
API reference	Sentence Transformers
GitHub link	https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/sentence_transformers
Package name	sentence-transformers-haystack

Overview

The SentenceTransformersDiversityRanker uses a ranking algorithm to order documents to maximize their overall diversity. It ranks a list of documents based on their similarity to the query. The component embeds the query and the documents using a pre-trained Sentence Transformers model.

This Ranker’s default model is sentence-transformers/all-MiniLM-L6-v2.

You can optionally set the top_k parameter, which specifies the maximum number of documents to return. It defaults to 10.

Authentication with a Hugging Face API token is only required to access private or gated models. You can pass the token at initialization with token, or set the HF_API_TOKEN or HF_TOKEN environment variable.

Find the full list of optional initialization parameters in our API reference.

Usage

Install the sentence-transformers-haystack package to use the SentenceTransformersDiversityRanker:

shell

pip install sentence-transformers-haystack

On its own

python

from haystack import Document
from haystack_integrations.components.rankers.sentence_transformers import (
    SentenceTransformersDiversityRanker,
)

ranker = SentenceTransformersDiversityRanker(
    model="sentence-transformers/all-MiniLM-L6-v2",
    similarity="cosine",
)

docs = [
    Document(content="Regular Exercise"),
    Document(content="Balanced Nutrition"),
    Document(content="Positive Mindset"),
    Document(content="Eating Well"),
    Document(content="Doing physical activities"),
    Document(content="Thinking positively"),
]

query = "How can I maintain physical fitness?"
output = ranker.run(query=query, documents=docs)
docs = output["documents"]

print(docs)

In a pipeline

python

from haystack import Document, Pipeline
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack_integrations.components.rankers.sentence_transformers import (
    SentenceTransformersDiversityRanker,
)

docs = [
    Document(content="The iconic Eiffel Tower is a symbol of Paris"),
    Document(content="Visit Luxembourg Gardens for a haven of tranquility in Paris"),
    Document(
        content="The Point Alexandre III bridge in Paris is famous for its Beaux-Arts style",
    ),
]
document_store = InMemoryDocumentStore()
document_store.write_documents(docs)

retriever = InMemoryBM25Retriever(document_store=document_store)
ranker = SentenceTransformersDiversityRanker()

document_ranker_pipeline = Pipeline()
document_ranker_pipeline.add_component(instance=retriever, name="retriever")
document_ranker_pipeline.add_component(instance=ranker, name="ranker")

document_ranker_pipeline.connect("retriever.documents", "ranker.documents")

query = "Most famous iconic sight in Paris"
document_ranker_pipeline.run(
    data={
        "retriever": {"query": query, "top_k": 3},
        "ranker": {"query": query, "top_k": 2},
    },
)