VespaDocumentStore

API reference	Vespa
GitHub link	https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/vespa

Vespa is an open-source big data serving engine that supports structured, text, and vector search at scale. The VespaDocumentStore connects Haystack to an existing Vespa application through pyvespa and supports both lexical and dense vector retrieval as well as metadata filtering.

Unlike most other Haystack Document Stores, the VespaDocumentStore does not create or deploy the Vespa application or schema for you. You configure Vespa with the fields and rank profiles you need, deploy it (either self-hosted or on Vespa Cloud), and then point the Document Store at the running endpoint.

Installation

Install the vespa-haystack integration:

shell

pip install vespa-haystack

To run Vespa locally, see the Vespa quick start. To deploy a managed Vespa application, see Vespa Cloud.

Usage

Prerequisites: Vespa Schema

Before using the VespaDocumentStore, you need a deployed Vespa application with a schema compatible with the fields you configure on the Document Store. By default, the integration expects:

• A text field named content for the Document body.
• A tensor field named embedding for dense vectors (when using embedding retrieval).
• A rank profile named bm25 for lexical retrieval (used by VespaKeywordRetriever).
• A rank profile named semantic that ranks with closeness(field, embedding) (used by VespaEmbeddingRetriever).

Field and rank profile names can be customized via the Document Store and Retriever constructors. See the Vespa documentation for details on writing schemas and rank profiles.

Authentication

The VespaDocumentStore supports the authentication methods provided by pyvespa:

• No authentication for local development against an unsecured Vespa endpoint.
• mTLS with a data plane certificate and key (via the cert and key parameters as Secrets).
• Bearer token for Vespa Cloud token endpoints (via vespa_cloud_secret_token or the VESPA_CLOUD_SECRET_TOKEN environment variable).

The Vespa endpoint URL can be passed via the url parameter or the VESPA_URL environment variable:

shell

export VESPA_URL="http://localhost"

For Vespa Cloud token authentication:

shell

export VESPA_URL="https://my-app.my-tenant.aws-us-east-1c.z.vespa-app.cloud"
export VESPA_CLOUD_SECRET_TOKEN="my-secret-token"

Initialization

Point the VespaDocumentStore at your deployed Vespa application and write Documents to it. The HTTP client is created lazily on first use:

python

from haystack import Document
from haystack_integrations.document_stores.vespa import VespaDocumentStore

document_store = VespaDocumentStore(
    url="http://localhost",
    schema="doc",
    namespace="doc",
    content_field="content",
    embedding_field="embedding",
    metadata_fields=["category"],
)

document_store.write_documents(
    [
        Document(
            content="Haystack integrates with Vespa for search.",
            meta={"category": "docs"},
        ),
        Document(
            content="Vespa supports lexical and vector retrieval.",
            meta={"category": "docs"},
        ),
    ],
)
print(document_store.count_documents())

To learn more about the initialization parameters, see our API docs.

To compute embeddings for your Documents, you can use a Document Embedder, such as the SentenceTransformersDocumentEmbedder.

Metadata Fields

Vespa is strictly schema-bound: every metadata field that you want to feed or read back from Vespa must exist as a field in the deployed schema. Use the metadata_fields parameter to declare an allowlist of metadata keys to send to Vespa on write and to request back on read. Metadata keys that are not in this allowlist are kept on Documents in memory but are not stored in Vespa.

Metadata Filtering

The VespaDocumentStore supports comparison operators (==, !=, >, >=, <, <=, in, not in) and the logical operators AND, OR, and NOT. Filters are translated to Vespa's YQL where clauses whenever possible.

Filters on date-typed values are evaluated client-side in Python when YQL cannot express the comparison directly. For more details on filter syntax, refer to Metadata Filtering.

Supported Retrievers

• VespaEmbeddingRetriever: A dense embedding-based Retriever that fetches Documents from Vespa using nearest-neighbor search and a configurable rank profile.
• VespaKeywordRetriever: A lexical Retriever that fetches Documents from Vespa using a configurable rank profile (BM25 by default).