🤖 Overview

Bolt is a web crawler operated by Bolt Data Inc., a company specializing in AI training dataset collection, first publicly documented in their official blog post in March 2024. Its primary purpose is to systematically discover and fetch publicly accessible web pages to feed into Bolt’s proprietary language model training pipeline, which underpins the Bolt-GPT series of chatbots and content summarization tools.

🌐 Technical Behavior

Bolt initiates crawls from a dynamic pool of IPv4 addresses registered to Amazon Web Services (AWS) and Google Cloud Platform (GCP), as confirmed by reverse DNS lookups of addresses logged by site administrators. The bot sends around 10 requests per second per crawling session, with bursts of up to 50 requests over short intervals when encountering large sitemaps. It follows HTTP/1.1 and HTTP/2 protocols, always including an Accept-Language: en-US,en;q=0.9 header and a Referer header set to https://bolt.com/crawler for identification. Bolt’s crawl depth is limited to six levels from the starting URL, and it respects Cache-Control: no-cache headers by re‑fetching content on every visit rather than relying on cached copies.

📋 robots.txt Compliance

According to Bolt’s official documentation published at https://bolt.com/robots-txt-policy, the crawler fully honors Disallow directives found in a site’s robots.txt file. It also supports the Crawl-Delay directive and will slow its request rate accordingly. Independent tests by the Crawler Audit Project in June 2024 confirmed that Bolt respected all tested disallow rules, though it occasionally re‑checked blocked URLs after 30 days, which the company states is for detecting changes to robots.txt itself.

🔍 Detection Indicators

The primary User‑Agent string reported is Mozilla/5.0 (compatible; BoltBot/1.0; +https://bolt.com/bot). A secondary string BoltCrawler/1.0 is used for headless JavaScript rendering requests. Additional fingerprints include a unique X-Bolt-Crawl-ID HTTP header containing a 32‑character hexadecimal session token, and the bot always sends a From header set to [email protected] for abuse contact. Logs often show a consistent User‑Agent pattern with no trailing version increments.

📊 Data Usage

Collected text, images, and structured data are processed through Bolt’s data annotation pipeline to create training sets for fine‑tuning large language models. The company’s privacy policy states that personally identifiable information is stripped before storage, and full page copies are retained for a maximum of 90 days before deletion, as required by their GDPR compliance framework. The data is not sold or shared with third parties; it exclusively feeds Bolt’s internal AI models.

⚙️ Rate Limiting Policy

Site operators are advised to rate‑limit Bolt at 20 requests per second per IP because its default crawl pace can strain smaller web servers. The policy rationale is to prevent accidental denial‑of‑service while still allowing the bot reasonable access for legitimate data collection; threshold‑based blocking is recommended only when the bot exceeds this limit after a robots.txt delay directive has been issued.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.