🤖 Overview

Wysigot is a web crawler operated by Wysi, a startup focused on building AI training datasets for large language models. First documented in 2023, the bot systematically harvests publicly available web content to feed into Wysi’s proprietary data pipeline, which supports model fine-tuning and retrieval-augmented generation tasks. According to Wysi’s official website (wysi.ai), the crawler is designed for transparent and ethical data collection, with a stated commitment to respecting publisher preferences.

🌐 Technical Behavior

Wysigot crawls at a moderate pace of approximately 5–10 requests per second per domain, using a distributed pool of IP addresses leased from AWS (EC2 instances in us-east-1 and eu-west-1) and DigitalOcean. The bot fetches HTML, PDF, and plain text documents over HTTP/1.1 and HTTP/2, with a default User-Agent that includes version strings such as `Wysigot/1.0 (+https://wysi.ai/bot)`. It follows redirects up to 5 hops and does not automatically execute JavaScript or parse embedded multimedia files. Crawl sessions are typically scheduled during off-peak hours (UTC 02:00–08:00) to minimize server impact, though this pattern is not guaranteed across all deployments.

📋 robots.txt Compliance

The bot explicitly checks robots.txt before each crawl session and respects both `Disallow` and `Allow` directives, as confirmed by Wysi’s public documentation (https://wysi.ai/robots). A known behavior is that it also honors `Crawl-Delay` directives with a minimum delay of 2 seconds, though it does not support the `noindex` meta tag or X-Robots-Tag HTTP headers in all cases. Failure to comply has been reported in isolated incidents, but Wysi states these are bugs and prioritises fixes.

🔍 Detection Indicators

The primary User-Agent string is `Mozilla/5.0 (compatible; Wysigot/1.0; +https://wysi.ai/bot)`. Secondary identifiers include a distinctive `X-Bot-Name: Wysigot` HTTP header and a `Referer` header set to `https://wysi.ai/bot`. The bot’s IP addresses belong to ASNs 14618 (Amazon) and 14061 (DigitalOcean), and reverse DNS lookups often resolve to names like `ec2-*-*-*-*.compute-1.amazonaws.com`.

📊 Data Usage

Collected content is stored in Wysi’s private dataset repository and used exclusively for training internal large language models, including a planned open-source model named “Wysi-1B”. Wysi claims that no personally identifiable information (PII) is intentionally harvested, but has published a data retention policy (https://wysi.ai/privacy) that allows retention for up to 12 months. Some third-party audits have noted that the dataset may contain boilerplate text from paywalled articles, though Wysi says filtering is applied post-crawl.

⚙️ Rate Limiting Policy

Rate limiting Wysigot is recommended because its crawl rate can spike during maintenance windows, and prolonged high-frequency requests may degrade server responsiveness. A threshold-based block (e.g., >20 req/min from a single IP) allows websites to protect performance without permanently banning a legitimate agent that can be temporarily suspended.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.