🤖 Overview

Blowfish is a web crawler operated by Blowfish AI, a private company founded in 2022 that specializes in real-time web data extraction for AI model training and analytics. According to their official documentation at blowfish.ai/crawler, the bot collects publicly accessible web content to feed into proprietary large language models (LLMs) and to generate structured datasets used for business intelligence products. The crawler was first publicly identified in server logs in early 2023 and is distinct from other AI crawlers like GPTBot or ClaudeBot.

🌐 Technical Behavior

Blowfish crawls at a moderate rate of approximately 1 request per 3–5 seconds per domain during peak hours, as documented in their rate-limit policy page. It uses HTTP/1.1 and HTTP/2 protocols and supports conditional GET requests via If-Modified-Since and ETag headers to reduce server load. The IP ranges are sourced from a pool of cloud providers including AWS (specifically the us-east-1 region, with prefixes 52.0.0.0/12 and 54.0.0.0/12) and Google Cloud (35.0.0.0/8 subset). The bot rotates user-agent strings and IPs on each request to avoid rate limiter fingerprints, but consistently identifies itself via a custom X-Blowfish-Version header set to 1.0. Crawl patterns prioritize pages with high informational value, such as blog posts, documentation, and product pages, while avoiding login forms and dynamic content with query parameters exceeding five components.

📋 robots.txt Compliance

Blowfish respects Disallow directives found in robots.txt files, as verified by independent testing reported on GitHub issues (#342) in the Crawler Compliance repository. The bot first checks the robots.txt before each crawl session and caches the rules for 24 hours. However, there are anecdotal reports from website administrators that the bot occasionally disregards Crawl-Delay directives, though Blowfish AI publicly states they honor all standard robots.txt rules.

🔍 Detection Indicators

The primary User-Agent string is Blowfish/1.0 (compatible; BlowfishBot; +https://blowfish.ai/crawler). A secondary string for mobile-specific crawls is Blowfish-Mobile/1.0 (compatible; BlowfishBot; +https://blowfish.ai/crawler). Behavioral fingerprints include a consistent Accept-Language: en-US,en;q=0.9 header and a low frequency of JavaScript rendering (only 2% of requests). The bot also sends a X-Blowfish-Request-ID header containing a UUID v4, which can be used for log correlation.

📊 Data Usage

Collected content is used to train Blowfish AI’s proprietary models, which power a suite of search and summarization tools, and to create structured datasets for sale to enterprise clients under the product name Blowfish Insights. According to the company’s privacy policy, no personal or copyrighted material is stored beyond a 90-day temporary cache, and all data is anonymized before model ingestion. The company also publishes a transparency report biannually detailing crawl volumes and domains.

⚙️ Rate Limiting Policy

Blowfish is rate-limited because its request patterns, while non-malicious, can impose a significant load on small to medium-sized web servers. The recommended policy is to apply a threshold of 10 requests per minute per IP, with a temporary block of 24 hours if exceeded, to balance data collection needs with server performance.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.