Parameters
Methods
read()
Scrapes or crawls the given URL, depending on mode, and returns a list of documents.
async_read()
Async variant of read(). Accepts the same parameters.Documentation Index
Fetch the complete documentation index at: /llms.txt
Use this file to discover all available pages before exploring further.
Parameters and methods of FirecrawlReader, which scrapes or crawls websites into documents using Firecrawl.
| Parameter | Type | Default | Description |
|---|---|---|---|
api_key | Optional[str] | None | Firecrawl API key for authentication |
params | Optional[Dict] | None | Additional parameters passed to the Firecrawl API |
mode | Literal["scrape", "crawl"] | "scrape" | "scrape" reads a single page, "crawl" follows links to read multiple pages |
chunk | bool | True | Chunk documents after reading |
chunk_size | int | 5000 | Size of each chunk when chunking is enabled |
chunking_strategy | Optional[ChunkingStrategy] | SemanticChunking() | Strategy used to chunk documents |
name | Optional[str] | None | Name of the reader |
description | Optional[str] | None | Description of the reader |
allowed_hosts | Optional[List[str]] | None | Hostnames the reader is allowed to fetch from. See Restricting URL Fetches. |
read()mode, and returns a list of documents.
| Parameter | Type | Default | Description |
|---|---|---|---|
url | str | Required | URL of the website to scrape or crawl |
name | Optional[str] | None | Name override for the resulting documents |
async_read()read(). Accepts the same parameters.Was this page helpful?