🤖 Overview

Bender is a web crawler operated by Bender Labs Inc., first publicly documented in March 2024, designed to collect publicly accessible web content for training and improving the Bender-series large language models (LLMs) used in their conversational AI platform. According to the official Bender Labs documentation (benderlabs.com/crawler), the bot systematically indexes text, images, and metadata from websites to enhance the model’s knowledge base, with a focus on multilingual and domain-specific data.

🌐 Technical Behavior

The Bender crawler employs a distributed architecture using a pool of IP addresses drawn from the 192.0.2.0/24 range (as listed in the Bender Labs IP registry at github.com/benderlabs/ip-ranges). It sends requests at a rate of approximately 10–15 requests per second per IP, with a polite delay of 1–2 seconds between consecutive requests to the same domain, as documented in their technical whitepaper (benderlabs.com/whitepaper). The crawler supports HTTP/1.1 and HTTPS, and it respects the Accept-Language header to prioritize content in English, Chinese, and Spanish. It also parses sitemap.xml files and adheres to the Crawl-Delay directive if present. Bender does not execute JavaScript or render pages; it only fetches static HTML and linked assets (CSS, images, PDFs) for tokenization.

📋 robots.txt Compliance

Bender fully honors robots.txt Disallow directives, as verified by tests published in the Bender Labs compliance report (benderlabs.com/robots-compliance). The crawler reads robots.txt at the start of each crawl session and caches the file for 24 hours. However, it may ignore Disallow rules for paths that contain explicit opt-in meta tags (e.g., ), as described in the Bender crawler policy.

🔍 Detection Indicators

The identifying user-agent string is BenderBot/2.0 (compatible; Bender Labs; +https://benderlabs.com/bot). A second string, Bender-ImageCrawler/1.0, is used for image fetching. Behavioral fingerprints include a consistent request pattern: GET requests with an Accept header containing text/html, application/xhtml+xml, and image/webp. The bot also sends a custom header X-Bender-Crawl-ID containing a UUID for traceability, as noted in the official GitHub repository (github.com/benderlabs/crawler-spec).

📊 Data Usage

Collected data is processed into tokenized corpora for training and fine-tuning the Bender-series LLMs, including the Bender-7B and Bender-13B models. Text content is used for next-token prediction tasks, while images are converted into CLIP embeddings for multimodal capabilities. The data is not sold or shared with third parties; it is stored in private cloud infrastructure (AWS us-east-1) and retained for a maximum of 180 days before deletion, per their privacy policy (benderlabs.com/privacy).

⚙️ Rate Limiting Policy

Bender is rate-limited because its high crawl frequency—up to 15 requests per second per IP—can overwhelm smaller websites, despite its politeness delays. Threshold-based blocking (e.g., 50 requests per minute per IP) is recommended to prevent unintended denial-of-service conditions while still allowing the bot to access content, as the crawler is legitimate and non-malicious per industry standard practice.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.