FirecrawlAgent
Crawler User-Agent:firecrawlagent
🤖 Overview
FirecrawlAgent is a legitimate web crawler operated by Firecrawl (formerly Firecrawl.dev), a Y Combinator‑backed startup founded by Eric Ciarla and Nicolas Silberstein. The agent is the default crawling engine for Firecrawl’s platform, which provides structured data extraction for developers building AI applications, LLM training pipelines, and knowledge graphs. FirecrawlAgent is designed to responsibly scrape publicly accessible websites on behalf of paying customers, feeding data into the Firecrawl API for use in retrieval‑augmented generation (RAG), fine‑tuning, and automated content monitoring. Official documentation at https://docs.firecrawl.dev describes it as “a fast, reliable web crawling API” that prioritizes ethical crawling practices.
🌐 Technical Behavior
FirecrawlAgent employs a headless Chromium browser behind the scenes, executing JavaScript to render modern single‑page applications (SPAs). It adheres to a maximum crawl depth of 10 levels by default and respects the robots.txt exclusions from the root domain. The crawler uses a rotating pool of IP addresses assigned to cloud providers including AWS, DigitalOcean, and Google Cloud. According to Firecrawl’s Github repository (https://github.com/nicklz/firecrawl), the default request rate is capped at 50 requests per minute per domain, with burst handling via exponential backoff. The agent sends an optional custom header X‑Firecrawl‑Agent: true and employs the standard User‑Agent: FirecrawlAgent/1.0 string. Connection uses HTTPS/2.0 with TLS 1.3 where the server supports it. FirecrawlAgent also supports sitemap‑based crawling and can be configured to preserve session cookies for authenticated content, though it will not automatically follow nofollow links.
📋 robots.txt Compliance
FirecrawlAgent explicitly respects robots.txt directives documented in the official docs (https://docs.firecrawl.dev/advanced/robots). The crawler reads the Disallow rules for the specific user‑agent “FirecrawlAgent” and also falls back to the wildcard “*” ruleset if no match is found. There is no evidence on the internet of FirecrawlAgent ignoring Crawl‑Delay or Disallow directives; the company states in its FAQ that “we never scrape pages disallowed by robots.txt.”
🔍 Detection Indicators
Primary detection relies on the User‑Agent string: FirecrawlAgent/1.0. Secondary fingerprints include the X‑Firecrawl‑Agent header, a fixed pattern of request intervals (commonly 1.2 seconds between GETs), and the use of Chrome/128.0 (headless) in the Accept‑Language header. The HTTP Sec‑CH‑UA header often reports “Firecrawl”. The agent’s IP ranges can be retrieved via the public API endpoint https://api.firecrawl.dev/v0/ip‑ranges, which dynamically lists the current CIDR blocks.
📊 Data Usage
Data collected by FirecrawlAgent is used exclusively by Firecrawl’s customers for AI training datasets, RAG pipelines, and internal analytics. Firecrawl’s privacy policy (https://firecrawl.dev/privacy) states that no raw scraped content is stored on Firecrawl servers after delivery; the data is forwarded directly to the customer’s storage endpoint. The platform also offers a “training mode” where content can be pre‑processed into embeddings for LLMs.
⚙️ Rate Limiting Policy
FirecrawlAgent is rate‑limited because it can initiate aggressive crawl patterns when processing large sitemaps. Web application owners should apply threshold‑based blocking (e.g., >30 requests in a sliding 10‑second window) to protect origin servers from accidental overload, while still allowing the agent to complete its legitimate indexing tasks within the published 50 req/min limit.
Similar Threats
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
