Pinecone

Module haystack_integrations.components.retrievers.pinecone.embedding_retriever

PineconeEmbeddingRetriever

Retrieves documents from the PineconeDocumentStore, based on their dense embeddings.

Usage example:

python

import os
from haystack.document_stores.types import DuplicatePolicy
from haystack import Document
from haystack import Pipeline
from haystack.components.embedders import SentenceTransformersTextEmbedder, SentenceTransformersDocumentEmbedder
from haystack_integrations.components.retrievers.pinecone import PineconeEmbeddingRetriever
from haystack_integrations.document_stores.pinecone import PineconeDocumentStore

os.environ["PINECONE_API_KEY"] = "YOUR_PINECONE_API_KEY"
document_store = PineconeDocumentStore(index="my_index", namespace="my_namespace", dimension=768)

documents = [Document(content="There are over 7,000 languages spoken around the world today."),
             Document(content="Elephants have been observed to behave in a way that indicates..."),
             Document(content="In certain places, you can witness the phenomenon of bioluminescent waves.")]

document_embedder = SentenceTransformersDocumentEmbedder()
document_embedder.warm_up()
documents_with_embeddings = document_embedder.run(documents)

document_store.write_documents(documents_with_embeddings.get("documents"), policy=DuplicatePolicy.OVERWRITE)

query_pipeline = Pipeline()
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
query_pipeline.add_component("retriever", PineconeEmbeddingRetriever(document_store=document_store))
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")

query = "How many languages are there?"

res = query_pipeline.run({"text_embedder": {"text": query}})
assert res['retriever']['documents'][0].content == "There are over 7,000 languages spoken around the world today."

PineconeEmbeddingRetriever.__init__

python

def __init__(*,
             document_store: PineconeDocumentStore,
             filters: dict[str, Any] | None = None,
             top_k: int = 10,
             filter_policy: str | FilterPolicy = FilterPolicy.REPLACE)

Arguments:

• document_store: The Pinecone Document Store.
• filters: Filters applied to the retrieved Documents.
• top_k: Maximum number of Documents to return.
• filter_policy: Policy to determine how filters are applied.

Raises:

• ValueError: If document_store is not an instance of PineconeDocumentStore.

PineconeEmbeddingRetriever.to_dict

python

def to_dict() -> dict[str, Any]

Serializes the component to a dictionary.

Returns:

Dictionary with serialized data.

PineconeEmbeddingRetriever.from_dict

python

@classmethod
def from_dict(cls, data: dict[str, Any]) -> "PineconeEmbeddingRetriever"

Deserializes the component from a dictionary.

Arguments:

• data: Dictionary to deserialize from.

Returns:

Deserialized component.

PineconeEmbeddingRetriever.run

python

@component.output_types(documents=list[Document])
def run(query_embedding: list[float],
        filters: dict[str, Any] | None = None,
        top_k: int | None = None) -> dict[str, list[Document]]

Retrieve documents from the PineconeDocumentStore, based on their dense embeddings.

Arguments:

• query_embedding: Embedding of the query.
• filters: Filters applied to the retrieved Documents. The way runtime filters are applied depends on the filter_policy chosen at retriever initialization. See init method docstring for more details.
• top_k: Maximum number of Documents to return.

Returns:

List of Document similar to query_embedding.

PineconeEmbeddingRetriever.run_async

python

@component.output_types(documents=list[Document])
async def run_async(query_embedding: list[float],
                    filters: dict[str, Any] | None = None,
                    top_k: int | None = None) -> dict[str, list[Document]]

Asynchronously retrieve documents from the PineconeDocumentStore, based on their dense embeddings.

Arguments:

• query_embedding: Embedding of the query.
• filters: Filters applied to the retrieved Documents. The way runtime filters are applied depends on the filter_policy chosen at retriever initialization. See init method docstring for more details.
• top_k: Maximum number of Documents to return.

Returns:

List of Document similar to query_embedding.

Module haystack_integrations.document_stores.pinecone.document_store

METADATA_SUPPORTED_TYPES

List[str] is supported and checked separately

PineconeDocumentStore

A Document Store using Pinecone vector database.

PineconeDocumentStore.__init__

python

def __init__(*,
             api_key: Secret = Secret.from_env_var("PINECONE_API_KEY"),
             index: str = "default",
             namespace: str = "default",
             batch_size: int = 100,
             dimension: int = 768,
             spec: dict[str, Any] | None = None,
             metric: Literal["cosine", "euclidean", "dotproduct"] = "cosine")

Creates a new PineconeDocumentStore instance.

It is meant to be connected to a Pinecone index and namespace.

Arguments:

• api_key: The Pinecone API key.
• index: The Pinecone index to connect to. If the index does not exist, it will be created.
• namespace: The Pinecone namespace to connect to. If the namespace does not exist, it will be created at the first write.
• batch_size: The number of documents to write in a single batch. When setting this parameter, consider documented Pinecone limits.
• dimension: The dimension of the embeddings. This parameter is only used when creating a new index.
• spec: The Pinecone spec to use when creating a new index. Allows choosing between serverless and pod deployment options and setting additional parameters. Refer to the Pinecone documentation for more details. If not provided, a default spec with serverless deployment in the us-east-1 region will be used (compatible with the free tier).
• metric: The metric to use for similarity search. This parameter is only used when creating a new index.

PineconeDocumentStore.close

python

def close()

Close the associated synchronous resources.

PineconeDocumentStore.close_async

python

async def close_async()

Close the associated asynchronous resources. To be invoked manually when the Document Store is no longer needed.

PineconeDocumentStore.from_dict

python

@classmethod
def from_dict(cls, data: dict[str, Any]) -> "PineconeDocumentStore"

Deserializes the component from a dictionary.

Arguments:

• data: Dictionary to deserialize from.

Returns:

Deserialized component.

PineconeDocumentStore.to_dict

python

def to_dict() -> dict[str, Any]

Serializes the component to a dictionary.

Returns:

Dictionary with serialized data.

PineconeDocumentStore.count_documents

python

def count_documents() -> int

Returns how many documents are present in the document store.

PineconeDocumentStore.count_documents_async

python

async def count_documents_async() -> int

Asynchronously returns how many documents are present in the document store.

PineconeDocumentStore.write_documents

python

def write_documents(documents: list[Document],
                    policy: DuplicatePolicy = DuplicatePolicy.NONE) -> int

Writes Documents to Pinecone.

Arguments:

• documents: A list of Documents to write to the document store.
• policy: The duplicate policy to use when writing documents. PineconeDocumentStore only supports DuplicatePolicy.OVERWRITE.

Returns:

The number of documents written to the document store.

PineconeDocumentStore.write_documents_async

python

async def write_documents_async(
        documents: list[Document],
        policy: DuplicatePolicy = DuplicatePolicy.NONE) -> int

Asynchronously writes Documents to Pinecone.

Arguments:

• documents: A list of Documents to write to the document store.
• policy: The duplicate policy to use when writing documents. PineconeDocumentStore only supports DuplicatePolicy.OVERWRITE.

Returns:

The number of documents written to the document store.

PineconeDocumentStore.filter_documents

python

def filter_documents(filters: dict[str, Any] | None = None) -> list[Document]

Returns the documents that match the filters provided.

For a detailed specification of the filters, refer to the documentation

Arguments:

• filters: The filters to apply to the document list.

Returns:

A list of Documents that match the given filters.

PineconeDocumentStore.filter_documents_async

python

async def filter_documents_async(
        filters: dict[str, Any] | None = None) -> list[Document]

Asynchronously returns the documents that match the filters provided.

Arguments:

• filters: The filters to apply to the document list.

Returns:

A list of Documents that match the given filters.

PineconeDocumentStore.delete_documents

python

def delete_documents(document_ids: list[str]) -> None

Deletes documents that match the provided document_ids from the document store.

Arguments:

• document_ids: the document ids to delete

PineconeDocumentStore.delete_documents_async

python

async def delete_documents_async(document_ids: list[str]) -> None

Asynchronously deletes documents that match the provided document_ids from the document store.

Arguments:

• document_ids: the document ids to delete

PineconeDocumentStore.delete_all_documents

python

def delete_all_documents() -> None

Deletes all documents in the document store.

PineconeDocumentStore.delete_all_documents_async

python

async def delete_all_documents_async() -> None

Asynchronously deletes all documents in the document store.

PineconeDocumentStore.delete_by_filter

python

def delete_by_filter(filters: dict[str, Any]) -> int

Deletes all documents that match the provided filters.

Pinecone does not support server-side delete by filter, so this method first searches for matching documents, then deletes them by ID.

Arguments:

• filters: The filters to apply to select documents for deletion. For filter syntax, see Haystack metadata filtering

Returns:

The number of documents deleted.

PineconeDocumentStore.delete_by_filter_async

python

async def delete_by_filter_async(filters: dict[str, Any]) -> int

Asynchronously deletes all documents that match the provided filters.

Pinecone does not support server-side delete by filter, so this method first searches for matching documents, then deletes them by ID.

Arguments:

• filters: The filters to apply to select documents for deletion. For filter syntax, see Haystack metadata filtering

Returns:

The number of documents deleted.

PineconeDocumentStore.update_by_filter

python

def update_by_filter(filters: dict[str, Any], meta: dict[str, Any]) -> int

Updates the metadata of all documents that match the provided filters.

Pinecone does not support server-side update by filter, so this method first searches for matching documents, then updates their metadata and re-writes them.

Arguments:

• filters: The filters to apply to select documents for updating. For filter syntax, see Haystack metadata filtering
• meta: The metadata fields to update. This will be merged with existing metadata.

Returns:

The number of documents updated.

PineconeDocumentStore.update_by_filter_async

python

async def update_by_filter_async(filters: dict[str, Any],
                                 meta: dict[str, Any]) -> int

Asynchronously updates the metadata of all documents that match the provided filters.

Pinecone does not support server-side update by filter, so this method first searches for matching documents, then updates their metadata and re-writes them.

Arguments:

• filters: The filters to apply to select documents for updating. For filter syntax, see Haystack metadata filtering
• meta: The metadata fields to update. This will be merged with existing metadata.

Returns:

The number of documents updated.

PineconeDocumentStore.count_documents_by_filter

python

def count_documents_by_filter(filters: dict[str, Any]) -> int

Returns the count of documents that match the provided filters.

Note: Due to Pinecone's limitations, this method fetches documents and counts them. For large result sets, this is subject to Pinecone's TOP_K_LIMIT of 1000 documents.

Arguments:

• filters: The filters to apply to the document list. For filter syntax, see Haystack metadata filtering

Returns:

The number of documents that match the filters.

PineconeDocumentStore.count_documents_by_filter_async

python

async def count_documents_by_filter_async(filters: dict[str, Any]) -> int

Asynchronously returns the count of documents that match the provided filters.

Note: Due to Pinecone's limitations, this method fetches documents and counts them. For large result sets, this is subject to Pinecone's TOP_K_LIMIT of 1000 documents.

Arguments:

• filters: The filters to apply to the document list.

Returns:

The number of documents that match the filters.

PineconeDocumentStore.count_unique_metadata_by_filter

python

def count_unique_metadata_by_filter(
        filters: dict[str, Any], metadata_fields: list[str]) -> dict[str, int]

Counts unique values for each specified metadata field in documents matching the filters.

Note: Due to Pinecone's limitations, this method fetches documents and aggregates in Python. Subject to Pinecone's TOP_K_LIMIT of 1000 documents.

Arguments:

• filters: The filters to apply to select documents.
• metadata_fields: List of metadata field names to count unique values for.

Returns:

Dictionary mapping field names to counts of unique values.

PineconeDocumentStore.count_unique_metadata_by_filter_async

python

async def count_unique_metadata_by_filter_async(
        filters: dict[str, Any], metadata_fields: list[str]) -> dict[str, int]

Asynchronously counts unique values for each specified metadata field in documents matching the filters.

Note: Due to Pinecone's limitations, this method fetches documents and aggregates in Python. Subject to Pinecone's TOP_K_LIMIT of 1000 documents.

Arguments:

• filters: The filters to apply to select documents.
• metadata_fields: List of metadata field names to count unique values for.

Returns:

Dictionary mapping field names to counts of unique values.

PineconeDocumentStore.get_metadata_fields_info

python

def get_metadata_fields_info() -> dict[str, dict[str, str]]

Returns information about metadata fields and their types by sampling documents.

Note: Pinecone doesn't provide a schema introspection API, so this method infers field types by examining the metadata of documents stored in the index (up to 1000 documents).

Type mappings:

• 'text': Document content field
• 'keyword': String metadata values
• 'long': Numeric metadata values (int or float)
• 'boolean': Boolean metadata values

Returns:

Dictionary mapping field names to type information. Example:

python

{
    'content': {'type': 'text'},
    'category': {'type': 'keyword'},
    'priority': {'type': 'long'},
}

PineconeDocumentStore.get_metadata_fields_info_async

python

async def get_metadata_fields_info_async() -> dict[str, dict[str, str]]

Asynchronously returns information about metadata fields and their types by sampling documents.

Note: Pinecone doesn't provide a schema introspection API, so this method infers field types by examining the metadata of documents stored in the index (up to 1000 documents).

Type mappings:

• 'text': Document content field
• 'keyword': String metadata values
• 'long': Numeric metadata values (int or float)
• 'boolean': Boolean metadata values

Returns:

Dictionary mapping field names to type information. Example:

python

{
    'content': {'type': 'text'},
    'category': {'type': 'keyword'},
    'priority': {'type': 'long'},
}

PineconeDocumentStore.get_metadata_field_min_max

python

def get_metadata_field_min_max(metadata_field: str) -> dict[str, Any]

Returns the minimum and maximum values for a metadata field.

Supports numeric (int, float), boolean, and string (keyword) types:

• Numeric: Returns min/max based on numeric value
• Boolean: Returns False as min, True as max
• String: Returns min/max based on alphabetical ordering

Note: This method fetches all documents and computes min/max in Python. Subject to Pinecone's TOP_K_LIMIT of 1000 documents.

Arguments:

• metadata_field: The metadata field name to analyze.

Raises:

• ValueError: If the field doesn't exist or has no values.

Returns:

Dictionary with 'min' and 'max' keys.

PineconeDocumentStore.get_metadata_field_min_max_async

python

async def get_metadata_field_min_max_async(
        metadata_field: str) -> dict[str, Any]

Asynchronously returns the minimum and maximum values for a metadata field.

Supports numeric (int, float), boolean, and string (keyword) types:

• Numeric: Returns min/max based on numeric value
• Boolean: Returns False as min, True as max
• String: Returns min/max based on alphabetical ordering

Note: This method fetches all documents and computes min/max in Python. Subject to Pinecone's TOP_K_LIMIT of 1000 documents.

Arguments:

• metadata_field: The metadata field name to analyze.

Raises:

• ValueError: If the field doesn't exist or has no values.

Returns:

Dictionary with 'min' and 'max' keys.

PineconeDocumentStore.get_metadata_field_unique_values

python

def get_metadata_field_unique_values(metadata_field: str,
                                     search_term: str | None = None,
                                     from_: int = 0,
                                     size: int = 10) -> tuple[list[str], int]

Retrieves unique values for a metadata field with optional search and pagination.

Note: This method fetches documents and extracts unique values in Python. Subject to Pinecone's TOP_K_LIMIT of 1000 documents.

Arguments:

• metadata_field: The metadata field name to get unique values for.
• search_term: Optional search term to filter values (case-insensitive substring match).
• from_: Starting offset for pagination (default: 0).
• size: Number of values to return (default: 10).

Returns:

Tuple of (list of unique values, total count of matching values).

PineconeDocumentStore.get_metadata_field_unique_values_async

python

async def get_metadata_field_unique_values_async(
        metadata_field: str,
        search_term: str | None = None,
        from_: int = 0,
        size: int = 10) -> tuple[list[str], int]

Asynchronously retrieves unique values for a metadata field with optional search and pagination.

Note: This method fetches documents and extracts unique values in Python. Subject to Pinecone's TOP_K_LIMIT of 1000 documents.

Arguments:

• metadata_field: The metadata field name to get unique values for.
• search_term: Optional search term to filter values (case-insensitive substring match).
• from_: Starting offset for pagination (default: 0).
• size: Number of values to return (default: 10).

Returns:

Tuple of (list of unique values, total count of matching values).