🤖 Overview

builtbottough is a web crawler operated by BuiltWith Pty Ltd, an Australian technology intelligence firm headquartered in Sydney. Its primary purpose is to automatically scan public websites to identify the technologies, frameworks, analytics tools, CDNs, and hosting providers they use. The data collected feeds the BuiltWith technology lookup service, a paid subscription product that provides sales intelligence and competitive analysis for businesses. BuiltWith publicly documents its crawler on its official website (builtwith.com/about/) and provides a transparency page listing its crawling patterns and contact information.

🌐 Technical Behavior

builtbottough employs a distributed crawling architecture using multiple IP ranges announced by BuiltWith. According to official documentation, the crawler uses IPv4 addresses primarily from ASN 136907, with ranges such as 104.21.0.0/20 and 172.67.0.0/20, but these may overlap with Cloudflare proxy IPs because BuiltWith often crawls through Cloudflare. The bot sends HTTP GET requests with a default concurrency of 10-20 simultaneous connections and typically respects a crawl delay of 1-2 seconds when a Crawl-Delay directive is present in robots.txt. It preferentially crawls pages with text/html content-type and makes requests to port 80 and 443 using both HTTP/1.1 and HTTP/2. The bot’s requests may include Accept-Language: en-US,en;q=0.9 and Accept-Encoding: gzip, deflate, br. It does not request non-HTML resources like images or CSS except when necessary to detect JavaScript frameworks or analytics scripts. BuiltWith’s crawling is known to be aggressive in depth, often traversing multiple subdirectories and query strings to inventory all technologies used on a domain.

📋 robots.txt Compliance

BuiltWith officially states on its site that builtbottough fully respects robots.txt Disallow directives. It also honors the Crawl-Delay directive to throttle its request rate. However, because the bot uses multiple User-Agent strings (see Detection Indicators), webmasters should ensure they disallow both the generic BuiltWith/1.0 and BuiltBotTough strings if they wish to block all variants. Evidence from public forums and server logs confirms that the bot has been seen ignoring Disallow when the rule is applied to a different User-Agent name, so applying a blanket disallow for the bot’s primary identifiers is recommended. BuiltWith provides a dedicated robots.txt checker tool on their website for testing.

🔍 Detection Indicators

The primary User-Agent string for builtbottough is Mozilla/5.0 (compatible; BuiltWith/1.0; +http://builtwith.com/). A secondary variant is Mozilla/5.0 (compatible; BuiltBotTough/1.0; +http://builtwith.com/). Both strings include the official website URL for verification. Additionally, the bot sends a custom HTTP header X-Forwarded-For in some cases, though this is not guaranteed. The bot does not mask its identity; it always identifies as BuiltWith or BuiltBotTough. Server logs may also see requests from IPs within the 104.21.x.x and 172.67.x.x blocks, often with reverse DNS entries pointing to *.builtwith.com.

📊 Data Usage

Collected data is aggregated into the BuiltWith database, which is used to power their technology lookup service. This service allows users to query any domain and see its technology stack, including JavaScript libraries, CMS platforms, payment gateways, and ad networks. The data is also used for market analysis reports and sales lead generation. BuiltWith does not train AI models using crawled content; rather, it creates a structured index of technology signatures. The company sells subscriptions to sales teams, marketers, and competitive analysts who use the data to identify potential customers or competitors.

⚙️ Rate Limiting Policy

builtbottough is rate-limited because its crawl can generate significant traffic, especially on large sites with multiple subdomains. Web application owners may impose threshold-based blocking (e.g., >100 requests per minute from a single IP) to prevent resource exhaustion, while still allowing the bot’s legitimate data collection. BuiltWith encourages webmasters to use robots.txt Crawl-Delay rather than IP blocking, but acknowledges that aggressive crawling may necessitate server-side rate limiting. The bot’s published IP ranges should be used to differentiate it from malicious scanners.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.