🤖 Overview

woriobot is a web crawler operated by Woriobot Inc., a company specializing in training data collection for AI language models. First publicly documented in mid-2023, the bot systematically indexes publicly accessible web content to feed into proprietary machine learning pipelines for natural language processing tasks. Its primary product is a curated dataset used by third-party AI developers and researchers to improve model accuracy on diverse text sources. The bot is listed on the official website woriobot.com with a user-agent identifier and a robots.txt policy page.

🌐 Technical Behavior

woriobot employs a breadth-first crawling strategy, typically initiating requests from a single seed URL per domain and following links up to a configurable depth (default: 3). According to community observations and server logs shared on user-agents.net, the bot sends requests at an average rate of 5–10 per second per domain, with occasional bursts during initial discovery. It connects over both HTTP/1.1 and HTTP/2, and respects 304 Not Modified headers to avoid re-crawling unchanged content. IP ranges are not publicly documented, but reverse DNS lookups on observed addresses (e.g., 203.0.113.x) resolve to crawl.woriobot.com. The bot includes a Referer header pointing to the crawled page’s origin and a Via header identifying the crawler version.

📋 robots.txt Compliance

Official documentation at woriobot.com/robots states that woriobot honors the Disallow directive in robots.txt exactly as specified, including wildcard patterns. Empirical tests reported on Stack Overflow threads confirm that after adding a Disallow rule, no further requests from the bot were logged within 24 hours. The bot also supports the Crawl-Delay directive, though it does not require it by default.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; woriobot/2.0; +https://woriobot.com/bot). Secondary identifiers include the User-Agent field alone (no token after a slash) and a custom HTTP header X-Woriobot-Version: 2.0. The bot occasionally varies the Accept-Language header to en-US,en;q=0.9 and does not send cookies or JavaScript-related headers.

📊 Data Usage

All collected text is processed into a proprietary training corpus for large language models under a pay-per-dataset licensing model. The dataset excludes personally identifiable information through automated redaction filters and is used by AI firms to fine-tune conversational agents, summarization engines, and translation systems. Woriobot also offers a “crawl-on-request” service where customers define specific URL patterns to include.

⚙️ Rate Limiting Policy

woriobot is rate-limited by many operators because its default crawl frequency, while moderate, can still overwhelm shared hosting environments or sites with limited bandwidth. Threshold-based blocking (e.g., 429 Too Many Requests after 50 requests in 60 seconds) is recommended to protect server resources while still allowing legitimate crawling.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.