DocumentStore

👍

Use with Retrievers

By far the most common way to use a DocumentStore in Haystack is to fetch documents using a Retriever. You provide a DocumentStore as an argument when you initialize a Retriever.

Initialization

To use a DocumentStore in a pipeline, you must initialize it first. Initializing a new DocumentStore in Haystack is straightforward. Have a look at the instructions for different types of DocumentStores:

docker pull docker.elastic.co/elasticsearch/elasticsearch:7.9.2
docker run -d -p 9200:9200 -e "discovery.type=single-node" elasticsearch:7.9.2

document_store = ElasticsearchDocumentStore()

from haystack.document_stores import FAISSDocumentStore

document_store = FAISSDocumentStore()

from haystack.document_stores import FAISSDocumentStore

# First, initialize the document store:
document_store = FAISSDocumentStore(faiss_index_factory_str="Flat")

# Save the document store:
document_store.save(index_path="my_faiss_index.faiss")
# Saving the document store creates two files: my_faiss_index.faiss and my_faiss_index.json

from haystack.document_stores import FAISSDocumentStore

document_store = FAISSDocumentStore(faiss_index_factory_str="Flat")
document_store.save(index_path="data/my_index.faiss", config_path="data/my_config.json")

# Load the saved index into a new DocumentStore instance:
new_document_store = FAISSDocumentStore(faiss_index_path="my_faiss_index.faiss")

# or use the `load()` method to create a new DocumentStore instance: 
new_document_store = FAISSDocumentStore.load(index_path="my_faiss_index.faiss")

# Also, provide `config_path` parameter if you set it when calling the `save()` method: 
new_document_store = FAISSDocumentStore.load(index_path="data/my_index.faiss", config_path="data/my_config.json")

# Check if the DocumentStore is loaded correctly
assert new_document_store.faiss_index_factory_str == "Flat"

from haystack.document_stores import InMemoryDocumentStore
document_store = InMemoryDocumentStore()

pip install -e "git+https://github.com/deepset-ai/haystack-extras.git#egg=milvus_documentstore&subdirectory=stores/milvus-documentstore"

from haystack import Document
from milvus_documentstore import MilvusDocumentStore

ds = MilvusDocumentStore()
ds.write_documents([Document("Some Content")])
ds.get_all_documents()  # prints [<Document: {'content': 'foo', 'content_type': 'text', ...>]

docker pull opensearchproject/opensearch:1.0.1
docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" opensearchproject/opensearch:1.0.1

from haystack.document_stores import OpenSearchDocumentStore
document_store = OpenSearchDocumentStore()

from haystack.document_stores import PineconeDocumentStore

document_store = PineconeDocumentStore(
    api_key='YOUR_API_KEY',
    similarity="cosine",
    index='your_index_name',
    embedding_dim=768
)

pip install qdrant-haystack

from qdrant_haystack.document_stores import QdrantDocumentStore

document_store = QdrantDocumentStore(
    ":memory:",
    index="Document",
    embedding_dim=512,
    recreate_index=True,
    hnsw_config={"m": 16, "ef_construct": 64}  # Optional
)

from haystack.document_stores import SQLDocumentStore
document_store = SQLDocumentStore()

docker run -d -p 8080:8080 --env AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED='true' --env PERSISTENCE_DATA_PATH='/var/lib/weaviate' semitechnologies/weaviate:1.17.2

from haystack.document_stores import WeaviateDocumentStore
document_store = WeaviateDocumentStore()

Input Format

Cast your data into Document objects before writing into a DocumentStore:

from haystack.document_stores import ElasticsearchDocumentStore

document_store = ElasticsearchDocumentStore()
documents = [
    Document(
      'content'=DOCUMENT_TEXT_HERE,
      'meta'={'name': DOCUMENT_NAME, ...}
  	),
  	...
]
document_store.write_documents(documents)

You can also cast it into dictionaries:

from haystack.document_stores import ElasticsearchDocumentStore

document_store = ElasticsearchDocumentStore()
dicts = [
    {
        'content': DOCUMENT_TEXT_HERE,
        'meta': {'name': DOCUMENT_NAME, ...}
    }, ...
]
document_store.write_documents(dicts)

Writing Documents (Sparse Retrievers)

Haystack allows you to write store documents in an optimized fashion so that query times can be kept low. For sparse, keyword-based retrievers such as BM25 and TF-IDF, you simply have to call DocumentStore.write_documents(). The creation of the inverted index which optimizes querying speed is handled automatically.

document_store.write_documents(dicts)

Writing Documents (Dense Retrievers)

For dense neural network-based retrievers like Dense Passage Retrieval, or Embedding Retrieval, indexing involves computing the Document embeddings which will be compared against the Query embedding.

The storing of the text is handled by DocumentStore.write_documents() and the computation of the embeddings is started by DocumentStore.update_embeddings().

document_store.write_documents(dicts)
document_store.update_embeddings(retriever)

This step is computationally intensive since it will engage the transformer-based encoders. Having GPU acceleration will significantly speed this up.

Approximate Nearest Neighbors Search

Approximate nearest neighbors (ANN) search brings significant improvements in document retrieval speed. It does so by approximating embedding similarity calculations which in turn brings a slight tradeoff in retrieval accuracy. You can use ANN in cases where documents have embeddings and where the collection of documents is sufficiently large. In our benchmarks, we found significant speed improvements when working with document collections as small as 10 thousand documents.

ANN is available in:

• FAISSDocumentStore
• OpenSearchDocumentStore
• WeaviateDocumentStore
• PineconeDocumentStore

For most of these DocumentStores, the class constructor has an index_type argument that you can set to turn ANN on or off. Each of these DocumentStores has support for the HNSW algorithm which you can learn more about in Hierarchical Navigable Small Worlds. See the API documentation of the specific DocumentStore you are using for more information on its ANN parameters.

Choosing the Right Document Store

The Document Stores have different characteristics. You should choose one depending on the maturity of your project, the use case, and the technical environment:

👍

Our Recommendations

Restricted environment: Use the InMemoryDocumentStore, if you are just giving Haystack a quick try on a small sample and are working in a restricted environment that complicates running Elasticsearch or other databases.

Allrounder: Use the ElasticSearchDocumentStore, if you want to evaluate the performance of different retrieval options (dense vs. sparse) and are aiming for a smooth transition from PoC to production.

Working with Existing Databases

If you have an existing Elasticsearch or OpenSearch database with indexed documents, you can very quickly make a Haystack compliant version using our elasticsearch_index_to_document_store or open_search_index_to_document_store function.

from haystack.document_stores import elasticsearch_index_to_document_store

new_ds = elasticsearch_index_to_document_store(
    document_store=empty_document_store,
    original_content_field="content",
    original_index_name="document",
    original_name_field="title",
    preprocessor=preprocessor,
    port=9201,
    verify_certs=False,
    scheme="https",
    username="admin",
    password="admin"
)

from haystack.document_stores import open_search_index_to_document_store

new_ds = open_search_index_to_document_store(
    document_store=empty_document_store,
    original_content_field="content",
    original_index_name="document",
    original_name_field="title",
    preprocessor=preprocessor,
    port=9201,
    verify_certs=False,
    scheme="https",
    username="admin",
    password="admin"
)