🤖 Overview

Webster is a web crawler operated by Webster AI, a company that provides large-scale internet data collection services for training natural language processing models. According to the official Webster AI documentation at webster.ai/crawler, the bot’s primary purpose is to index publicly accessible web pages and feeds harvested text, metadata, and structural data into proprietary datasets used for improving the company’s language models. Unlike general-purpose search engine crawlers, Webster focuses on high-quality, diverse content to supplement training corpora.

🌐 Technical Behavior

Webster employs both HTTP/1.1 and HTTP/2 protocols, with a default crawl rate of 5 requests per second per source IP, as specified in the company’s technical white paper (Webster AI, 2023). The crawler cycles through a pool of IPv4 addresses allocated in the 192.0.2.0/24 range (documented in the bot’s IP reputation list) and uses a randomized user‑agent token to avoid fingerprinting. It supports ETags and If‑Modified‑Since headers to minimize bandwidth waste on unchanged content. The crawler’s request pattern includes both HEAD and GET requests, and it respects the Crawl‑Delay directive when present in robots.txt.

📋 robots.txt Compliance

Webster’s official guidelines at webster.ai/robots explicitly state that the bot fully honors Disallow and Allow directives. The crawler checks robots.txt files at least every 24 hours and caches them for the duration of a crawl session. A community audit published on GitHub (github.com/webster-ai/crawler/issues/42) confirmed that Webster does not follow links or resources that are disallowed, and it respects wildcard patterns such as Disallow: /private/.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; Webster/1.0; +http://webster.ai/bot); a secondary variant for mobile content is Mozilla/5.0 (Linux; Android 10; Webster/1.0). Webster also sends a custom HTTP header X‑Webster‑Bot: true and a From header containing the email address [email protected]. Behavioral fingerprinting reveals it always requests robots.txt before any other resource and has a median inter‑request interval of 200 milliseconds.

📊 Data Usage

Data collected by Webster is used exclusively for training the company’s Webster‑LLM series of large language models, as described in their model card (webster.ai/models/llm‑v2). The crawled content – including articles, forum posts, and code repositories – is transformed into tokenized training examples after removing personally identifiable information. Webster AI also publishes a public dataset summary on their research page, noting that approximately 60% of the collected data comes from English‑language sites.

⚙️ Rate Limiting Policy

Because Webster can sustain high request volumes when crawling new domains, web administrators are advised to enforce rate limits at the server level. The recommended threshold is 100 requests per minute per IP, after which the bot will respect a 503 response and back off for at least 10 minutes. This policy balances the need for comprehensive data collection with the preservation of server resources and user experience.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.