DocumentCleaner
Use DocumentCleaner
to make text documents more readable. It removes extra whitespaces, empty lines, specified substrings, regexes, page headers, and footers in this particular order. This is useful for preparing the documents for further processing by LLMs.
Name | DocumentCleaner |
Folder Path | /preprocessors/ |
Position in a Pipeline | In indexing pipelines after File Converters, before DocumentSplitter |
Inputs | "documents": a list of Documents |
Outputs | "documents": a list of Documents |
Overview
DocumentCleaner
expects a list of Documents as input and returns a list of Documents with cleaned texts. Selectable cleaning steps for each input Document are to remove_empty_lines
, remove_extra_whitespaces
and to remove_repeated_substrings
. These three parameters are booleans that can be set when the component is initialized.
remove_empty_lines
removes empty lines from the Document.remove_extra_whitespaces
removes extra whitespaces from the Document.remove_repeated_substrings
removes repeated substrings (headers/footers) from pages in the Document. Pages in the text need to be separated by form feed character "\f", which is supported byTextFileToDocument
andAzureOCRDocumentConverter
.
In addition, you can specify a list of strings that should be removed from all Documents as part of the cleaning with the parameter remove_substring
. You can also specify a regular expression with the parameter remove_regex
and any matches will be removed.
The cleaning steps are executed in the following order:
- remove_extra_whitespaces
- remove_empty_lines
- remove_substrings
- remove_regex
- remove_repeated_substrings
Usage
On its own
You can use it outside of a pipeline to clean up your Documents:
from haystack.components.preprocessors import DocumentCleaner
cleaner = DocumentCleaner(
remove_empty_lines=True,
remove_extra_whitespaces=True,
remove_repeated_substrings=False)
In a pipeline
from haystack import Document
from haystack import Pipeline
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.converters import TextFileToDocument
from haystack.components.preprocessors import DocumentCleaner
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.writers import DocumentWriter
document_store = InMemoryDocumentStore()
p = Pipeline()
p.add_component(instance=TextFileToDocument(), name="text_file_converter")
p.add_component(instance=DocumentCleaner(), name="cleaner")
p.add_component(instance=DocumentSplitter(split_by="sentence", split_length=1), name="splitter")
p.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
p.connect("text_file_converter.documents", "cleaner.documents")
p.connect("cleaner.documents", "splitter.documents")
p.connect("splitter.documents", "writer.documents")
p.run({"cleaner": {"documents": your_docs}})
Updated 8 months ago