OtariRanker

This component ranks documents by their relevance to the query using rerank models from many providers through an Otari LLM gateway.

Most common position in a pipeline	In a query pipeline, after a component that returns a list of documents such as a Retriever
Mandatory init variables	api_key: An Otari API key. Can be set with OTARI_API_KEY env var.
Mandatory run variables	query: A query string documents: A list of document objects
Output variables	documents: A list of document objects meta: A dictionary with the model used and usage information
API reference	Otari
GitHub link	https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/otari
Package name	otari-haystack

Overview

Otari is an open-source, OpenAI-compatible LLM gateway by mozilla.ai. It routes requests to over 40 model providers and adds API keys, budgets, and usage and cost tracking.

OtariRanker reranks documents through the gateway's /rerank endpoint. Use it after a Retriever in a query pipeline to reorder the retrieved documents by relevance to the query. Name the model with a selector that the gateway accepts, such as cohere:rerank-v3.5 (default).

By default, OtariRanker calls otari.ai at https://api.otari.ai/api/v1. For an account in otari.ai's EU region, whose API keys start with otk_v1_eu_, set api_base_url to https://eu.api.otari.ai/api/v1. For a gateway you run yourself, set api_base_url to its address followed by /api/v1, for example http://localhost:8000/api/v1.

OtariRanker needs an Otari API key to work. It uses an OTARI_API_KEY environment variable by default. Otherwise, you can pass an API key at initialization with api_key using Haystack's Secret API.

You can also specify the top_k parameter to set the maximum number of documents to return, and the score_threshold parameter to drop documents with a relevance score below a given value. To limit how much of each document the model reads, set max_tokens_per_doc.

Requests that time out, fail to connect, or get a rate-limit or temporary error from the gateway are retried. Set timeout and max_retries to change the defaults of 30 seconds and 3 retries.

Provider credentials

The gateway needs a credential for a provider that serves rerank models, such as Cohere. On otari.ai, add a provider key on your organization's Providers page. On a gateway you run yourself, add the provider on the Providers page of its dashboard, or declare it in the gateway's config.yml:

yaml

providers:
  cohere:
    api_key: ${COHERE_API_KEY}

The gateway reads a provider's environment variable, such as COHERE_API_KEY, only if its config names the variable like this. For details, see Configuring a provider in the Otari docs.

Embedding meta fields

Some use cases benefit from including meta information (such as a title) alongside the document content when reranking. Pass the names of the meta fields to include through the meta_fields_to_embed parameter; they will be concatenated with the document content using meta_data_separator.

python

from haystack_integrations.components.rankers.otari import OtariRanker

ranker = OtariRanker(meta_fields_to_embed=["title"], meta_data_separator="\n")

Usage

Install the otari-haystack package to use the OtariRanker:

shell

pip install otari-haystack

On its own

python

from haystack import Document
from haystack_integrations.components.rankers.otari import OtariRanker

ranker = OtariRanker()

docs = [
    Document(content="The capital of Brazil is Brasilia."),
    Document(content="The capital of France is Paris."),
]
result = ranker.run(query="What is the capital of France?", documents=docs)
print(result["documents"][0].content)
# >> The capital of France is Paris.

In a pipeline

python

from haystack import Document, Pipeline
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack_integrations.components.rankers.otari import OtariRanker

docs = [
    Document(content="Paris is in France"),
    Document(content="Berlin is in Germany"),
    Document(content="Lyon is in France"),
]
document_store = InMemoryDocumentStore()
document_store.write_documents(docs)

retriever = InMemoryBM25Retriever(document_store=document_store)
ranker = OtariRanker()

document_ranker_pipeline = Pipeline()
document_ranker_pipeline.add_component(instance=retriever, name="retriever")
document_ranker_pipeline.add_component(instance=ranker, name="ranker")

document_ranker_pipeline.connect("retriever.documents", "ranker.documents")

query = "Cities in France"
result = document_ranker_pipeline.run(
    data={
        "retriever": {"query": query, "top_k": 3},
        "ranker": {"query": query, "top_k": 2},
    },
)

print(result["ranker"]["documents"][0])
# >> Document(id=..., content: 'Paris is in France', score: ...)