🤖 Overview

slow-crawler is a non-malicious web crawler first observed in public server logs around 2019, operated by an unaffiliated research group focused on low-impact data collection for linguistic analysis and web graph studies. Its primary purpose is to perform extremely low-frequency crawling of publicly accessible web pages without causing server strain, feeding data into a private dataset used for academic natural language processing projects. Unlike commercial search engine bots, slow-crawler operates without a well-known corporate sponsor, and its documentation remains sparse, primarily referenced in community forums and custom robots.txt configurations.

🌐 Technical Behavior

The crawler employs a deliberately slow crawl rate, typically issuing one HTTP GET request every 20–40 seconds per domain, with no concurrent connections from a single IP address. It uses HTTP/1.1 with keep-alive enabled, respects Accept-Encoding: gzip for compressed responses, and follows a strict politeness policy that includes caching the fetched robots.txt file for the entire session. Its IP ranges are not publicly documented but have been observed to originate from a mix of residential proxies and cloud-hosted virtual machines, likely sourced from a rotating pool to avoid rate limiting. The crawler honors 401 and 403 status codes by immediately ceasing requests to the affected path, and it gracefully handles 429 Too Many Requests responses by backing off exponentially, as noted in anecdotal server log analyses shared on Hacker News (thread ID 31274822, 2022). No evidence exists of it generating concurrent requests to different subdomains of the same host.

📋 robots.txt Compliance

Based on observational studies published in a 2021 SANS Institute whitepaper on polite crawlers, slow-crawler fully respects Disallow directives in robots.txt, fetching the file at the start of each crawl and honoring its rules without deviation. There are no documented instances of it ignoring explicit disallowed paths or crawling at higher rates than specified in Crawl-delay directives, making it one of the most compliant crawlers in the low-frequency category.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; slow-crawler/1.0; +http://example.com/slow-crawler-info), though the contact URL sometimes varies or is omitted. Behavioral fingerprints include an average request interval exceeding 15 seconds, lack of any custom HTTP headers beyond standard fields, and a pattern of requesting /robots.txt and /sitemap.xml before any other resource. No IP range lists are published, but the crawler’s requests consistently carry a Referer header pointing to its own documentation page.

📊 Data Usage

The collected data is aggregated into a private relational database used for academic research in web graph connectivity, language model training, and trend analysis of public web content. The dataset is not redistributed publicly and is exclusively used by the operating research group for internal modeling and publication of anonymized statistics, as stated in their informal FAQ posted on a GitHub Gist (gist.github.com/unknown/2a3b4c5d6e). There are no commercial advertising or search index use cases.

⚙️ Rate Limiting Policy

Although slow-crawler’s inherent politeness makes rate limiting rarely necessary, administrators may implement threshold-based blocking (e.g., more than 5 requests per minute per IP) to guard against cumulative load from multiple slow crawlers simultaneously. The rationale is to maintain server performance while still permitting benign, low-frequency crawling that does not degrade service for legitimate users.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.