🤖 Overview

Bravebot is a web crawler operated by Brave Software, Inc., the company behind the privacy-focused Brave browser and Brave Search. First publicly documented in 2021, Bravebot's primary purpose is to index publicly accessible web pages to power Brave Search, an independent search engine that does not rely on Google or Bing. The crawler collects content to populate search results while respecting user privacy and publisher preferences. Brave Software explicitly states that Bravebot does not collect personally identifiable information or track users across sessions.

🌐 Technical Behavior

Bravebot crawls the web using standard HTTP/1.1 and HTTP/2 protocols, typically sending requests from IP addresses within Brave's autonomous system (AS16876). The bot respects the Crawl-Delay directive in robots.txt, defaulting to a delay of 1 second between requests unless a site specifies otherwise. It uses a moderate crawl rate, generally not exceeding a few requests per second per host, and identifies itself via the User-Agent string Bravebot/1.0 or a full string like Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Bravebot/1.0; +https://brave.com/bravebot/. Brave also publishes a list of IP ranges used by the crawler on its official documentation page at https://brave.com/bravebot/.

📋 robots.txt Compliance

Bravebot fully honors robots.txt directives, including Disallow, Allow, and Crawl-Delay rules. The official Bravebot page states that the bot abides by the Robots Exclusion Protocol and will respect site owner preferences regarding what content can be crawled. There is no documented evidence of Bravebot ignoring robots.txt; it consistently adheres to the standard to maintain good relations with webmasters.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string: Bravebot/1.0 or the fully qualified Mozilla/5.0 (compatible; Bravebot/1.0; +https://brave.com/bravebot/). Bravebot also sends a From header with the email address [email protected] in some configurations, though this is not guaranteed. Reverse DNS lookups on requesting IPs often resolve to hostnames under brave.com or bravebot.brave.com. No other custom HTTP headers are consistently sent; the bot behaves similarly to a standard modern browser in terms of accepted content types.

📊 Data Usage

Collected data is used exclusively for search indexing within Brave Search. Brave Software explicitly states that Bravebot does not collect data for training AI models or for advertising profiling. The crawled content is stored temporarily to build and update the search index, and the company emphasizes that no user data is sold or shared with third parties. Brave Search aims to provide a privacy-respecting alternative to mainstream search engines, and Bravebot's operation aligns with that mission.

⚙️ Rate Limiting Policy

While Bravebot is not malicious, its moderate crawl rate can still place load on smaller websites. Many webmasters choose to rate-limit Bravebot by setting a higher Crawl-Delay in robots.txt or by implementing server-side throttling. The policy rationale is to prevent any single crawler from consuming excessive resources, ensuring fair access for all visitors. Threshold-based blocking is applied only when the bot exceeds site-specific traffic limits, typically beyond what is considered normal for a well-behaved crawler.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.