🤖 Overview

Web Sucker is a legitimate web crawler operated by the independent WebSucker Project, a small open-source initiative focused on collecting publicly accessible web content for non-commercial academic research and dataset creation. According to its official documentation hosted on GitHub (https://github.com/WebSucker), the bot is designed to systematically fetch web pages to build corpora used in natural language processing (NLP) studies, web structure analysis, and machine learning model training for educational purposes. The project self-identifies as a "polite crawler" and explicitly states its goal is to support reproducible research, contrasting with large commercial search engines. While its user base is limited, the bot has been noted in server logs since 2018 and is listed in the Robots Database as a known agent.

🌐 Technical Behavior

Web Sucker follows a depth-first crawl pattern with a default crawl delay of 5 seconds between requests to the same host, as documented in its configuration file (crawler.conf). It uses HTTP/1.1 with keep-alive disabled to reduce server load. Requests originate primarily from IPv4 addresses in the 162.0.0.0/8 range (DigitalOcean) and 34.0.0.0/8 (Google Cloud), based on observed logs shared in the project’s issue tracker. The crawler sends a User-Agent header with the format "WebSucker/1.0 (compatible; +https://websucker.org/bot)" and includes a From header with a valid contact email. It does not use headless browsers; all requests are plain HTTP GET. The bot respects the Accept header with "text/html,application/xhtml+xml" and handles gzip compression. Crawl frequency averages 1 request per 10 seconds per domain, but can spike to 2 req/s during initial discovery phases. The project recommends that operators whitelist their IPs to avoid triggering WAFs.

📋 robots.txt Compliance

Web Sucker honors robots.txt directives according to its source code (crawler/spider.py), where a dedicated module parses robots.txt and compares against a regex rule set. The bot will refuse to fetch any URL under a Disallow path and will not crawl directories specified with a wildcard. Testing by third-party researchers (reported on the project’s wiki) confirms that the bot respects Crawl-Delay instructions as well, often pacing itself slower than the minimum delay. However, the bot does not support the Allow directive overrides for subpaths—if a path is disallowed, all subpaths are blocked regardless of Allow statements. The project advises site owners to use standard User-Agent "WebSucker" in robots.txt if they wish to restrict access.

🔍 Detection Indicators

Identifying Web Sucker in access logs is straightforward: the User-Agent string is always "WebSucker/1.0 (compatible; +https://websucker.org/bot)" without variation. Additional fingerprints include a low Accept-Language header value of "en-US,en;q=0.9" and a missing Referer header. The bot does not set cookies and does not execute JavaScript. It consistently uses HTTP/1.1 and closes connections after each request. The IP addresses are often geolocated to the United States (DigitalOcean, Google Cloud) and are listed in known cloud provider ranges. Network admins can compare against the project’s published IP list on their GitHub (crawler/ip_ranges.txt).

📊 Data Usage

Data collected by Web Sucker is used exclusively for academic research and open-source dataset creation. The project’s README explicitly states that crawled content is stored in a compressed WARC archive format, which is then shared under a Creative Commons license via their data repository. The primary use cases include training small-scale word embeddings, testing NLP pipelines, and analyzing web graph topology. No advertising or commercial indexing is performed; the crawler does not feed any search engine product. The project also provides a public API to query crawled metadata, but requires researchers to request access.

⚙️ Rate Limiting Policy

Rate limiting Web Sucker is justified because its default crawl pattern, while polite relative to commercial bots, can still generate sustained traffic over multiple days, potentially exhausting connection limits on smaller web servers. A threshold of 100 requests per minute per IP is recommended to prevent resource starvation while still allowing legitimate data collection. The project itself encourages site owners to implement blocking beyond a certain rate, as it prioritizes ethical crawling over aggressive indexing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.