PreProcessors
Use the PreProcessors to preprare your data normalize white spaces, remove headers and footers, clean empty lines in your Documents, or split them into smaller pieces. PreProcessors are useful in an indexing pipeline to prepare your files for search.
| PreProcessor | Description |
|---|---|
| CSVDocumentCleaner | Cleans CSV documents by removing empty rows and columns while preserving specific ignored rows and columns. |
| CSVDocumentSplitter | Divides CSV documents into smaller sub-tables based on empty rows and columns. |
| DocumentCleaner | Removes extra whitespaces, empty lines, specified substrings, regexes, page headers, and footers from documents. |
| DocumentSplitter | Splits a list of text documents into a list of text documents with shorter texts. |
| RecursiveSplitter | Splits text into smaller chunks, it does so by recursively applying a list of separators to the text, applied in the order they are provided. |
| TextCleaner | Removes regexes, punctuation, and numbers, as well as converts text to lowercase. Useful to clean up text data before evaluation. |
Updated 10 days ago
Related Links