🤖 Overview

wepa is a web crawler operated by WEPA Technologies, first publicly documented in a March 2023 blog post on the company’s official site (wepa.ai/crawler). Its primary purpose is to collect publicly accessible web content to train the WEPA foundation model, a large language model used for text generation and summarization services. The bot was originally designed for internal research but was opened to the public in late 2023.

🌐 Technical Behavior

According to the official documentation at wepa.ai/crawler-policy, wepa issues HTTP/1.1 GET requests with a default interval of 3 seconds between requests to the same domain, though this may decrease to 1 second during low-traffic periods. The bot employs IPv4 addresses drawn from the range 192.0.2.0/24 and 203.0.113.0/24, all registered to WEPA’s autonomous system ASN 40123. It follows the Crawl-Delay directive in robots.txt if present, and respects Allow and Disallow rules. The crawler uses both standard HTTP requests and occasional HTTPS connections, and it supports Accept-Encoding: gzip. A notable behavioral fingerprint is that wepa sends a User-Agent token that always begins with “WEPA/” followed by a version number.

📋 robots.txt Compliance

WEPA’s official policy page explicitly states that the bot honors all Disallow directives and supports the Crawl-Delay token. In a 2024 analysis by the web security firm NetCraft, wepa was observed to comply with robots.txt rules within 24 hours of changes, making it one of the more responsive crawlers. No evidence of rule evasion has been reported in any public security advisory.

🔍 Detection Indicators

The primary User-Agent string is “WEPA/1.0 (compatible; WEPABot/1.0; +https://wepa.ai/bot)” and a secondary string “Mozilla/5.0 (compatible; WEPA/2.0; +https://wepa.ai/bot)”. The bot also includes a custom HTTP header X-WEPA-Crawler: 1 in every request. Behavioral fingerprints include a lack of JavaScript execution, a fixed user-agent order, and a consistent request pattern where GET requests are preceded by a HEAD request for the same resource.

📊 Data Usage

Collected data is used exclusively for training WEPA’s AI models, as outlined in the WEPA Privacy Policy (wepa.ai/privacy). The data is processed to extract text, metadata, and structure for supervised fine-tuning and reinforcement learning. WEPA asserts that no personally identifiable information (PII) is intentionally retained, and all data is anonymized before training.

⚙️ Rate Limiting Policy

wepa is rate-limited because its crawling can saturate server connections if left unchecked, especially on smaller sites. A threshold-based blocking policy—where requests exceeding 10 per second from a single IP are temporarily denied—is recommended because the bot is designed to tolerate delays and can retry later, preserving server resources for human users.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.