Firecrawl

haystack_integrations.components.fetchers.firecrawl.firecrawl_crawler

FirecrawlCrawler

A component that uses Firecrawl to crawl one or more URLs and return the content as Haystack Documents.

Crawling starts from each given URL and follows links to discover subpages, up to a configurable limit. This is useful for ingesting entire websites or documentation sites, not just single pages.

Firecrawl is a service that crawls websites and returns content in a structured format (e.g. Markdown) suitable for LLMs. You need a Firecrawl API key from firecrawl.dev.

Usage example

python

from haystack_integrations.components.fetchers.firecrawl import FirecrawlCrawler

crawler = FirecrawlCrawler(
    api_key=Secret.from_env_var("FIRECRAWL_API_KEY"),
    params={"limit": 5},
)
crawler.warm_up()

result = crawler.run(urls=["https://docs.haystack.deepset.ai/docs/intro"])
documents = result["documents"]

init

python

__init__(
    api_key: Secret = Secret.from_env_var("FIRECRAWL_API_KEY"),
    params: dict[str, Any] | None = None,
) -> None

Initialize the FirecrawlCrawler.

Parameters:

• api_key (Secret) – API key for Firecrawl. Defaults to the FIRECRAWL_API_KEY environment variable.
• params (dict[str, Any] | None) – Parameters for the crawl request. See the Firecrawl API reference for available parameters. Defaults to {"limit": 1, "scrape_options": {"formats": ["markdown"]}}. Without a limit, Firecrawl may crawl all subpages and consume credits quickly.

run

python

run(urls: list[str], params: dict[str, Any] | None = None) -> dict[str, Any]

Crawls the given URLs and returns the extracted content as Documents.

Parameters:

• urls (list[str]) – List of URLs to crawl.
• params (dict[str, Any] | None) – Optional override of crawl parameters for this run. If provided, fully replaces the init-time params.

Returns:

• dict[str, Any] – A dictionary with the following keys:
• documents: List of documents, one for each URL crawled.

run_async

python

run_async(
    urls: list[str], params: dict[str, Any] | None = None
) -> dict[str, Any]

Asynchronously crawls the given URLs and returns the extracted content as Documents.

Parameters:

• urls (list[str]) – List of URLs to crawl.
• params (dict[str, Any] | None) – Optional override of crawl parameters for this run. If provided, fully replaces the init-time params.

Returns:

• dict[str, Any] – A dictionary with the following keys:
• documents: List of documents, one for each URL crawled.

warm_up

python

warm_up() -> None

Warm up the synchronous Firecrawl client.

warm_up_async

python

warm_up_async() -> None

Warm up the asynchronous Firecrawl client.

haystack_integrations.components.websearch.firecrawl.firecrawl_websearch

FirecrawlWebSearch

A component that uses Firecrawl to search the web and return results as Haystack Documents.

This component wraps the Firecrawl Search API, enabling web search queries that return structured documents with content and links. It follows the standard Haystack WebSearch component interface.

Firecrawl is a service that crawls and scrapes websites, returning content in formats suitable for LLMs. You need a Firecrawl API key from firecrawl.dev.

Usage example

python

from haystack_integrations.components.websearch.firecrawl import FirecrawlWebSearch
from haystack.utils import Secret

websearch = FirecrawlWebSearch(
    api_key=Secret.from_env_var("FIRECRAWL_API_KEY"),
    top_k=5,
)
result = websearch.run(query="What is Haystack by deepset?")
documents = result["documents"]
links = result["links"]

init

python

__init__(
    api_key: Secret = Secret.from_env_var("FIRECRAWL_API_KEY"),
    top_k: int | None = 10,
    search_params: dict[str, Any] | None = None,
) -> None

Initialize the FirecrawlWebSearch component.

Parameters:

• api_key (Secret) – API key for Firecrawl. Defaults to the FIRECRAWL_API_KEY environment variable.
• top_k (int | None) – Maximum number of documents to return. Defaults to 10. This can be overridden by the "limit" parameter in search_params.
• search_params (dict[str, Any] | None) – Additional parameters passed to the Firecrawl search API. See the Firecrawl API reference for available parameters. Supported keys include: tbs, location, scrape_options, sources, categories, timeout.

warm_up

python

warm_up() -> None

Warm up the synchronous Firecrawl client.

warm_up_async

python

warm_up_async() -> None

Warm up the asynchronous Firecrawl client.

run

python

run(query: str, search_params: dict[str, Any] | None = None) -> dict[str, Any]

Search the web using Firecrawl and return results as Documents.

Parameters:

• query (str) – Search query string.
• search_params (dict[str, Any] | None) – Optional override of search parameters for this run. If provided, fully replaces the init-time search_params.

Returns:

• dict[str, Any] – A dictionary with the following keys:
• documents: List of documents with search result content.
• links: List of URLs from the search results.

run_async

python

run_async(
    query: str, search_params: dict[str, Any] | None = None
) -> dict[str, Any]

Asynchronously search the web using Firecrawl and return results as Documents.

Parameters:

• query (str) – Search query string.
• search_params (dict[str, Any] | None) – Optional override of search parameters for this run. If provided, fully replaces the init-time search_params.

Returns:

• dict[str, Any] – A dictionary with the following keys:
• documents: List of documents with search result content.
• links: List of URLs from the search results.