🤖 Overview

SpankBot is an automated web crawler operated by SpankBang, a popular adult entertainment platform, first publicly documented in 2017. Its primary purpose is to index publicly accessible web pages—particularly video hosting sites and media platforms—to feed metadata and links into SpankBang’s search engine and recommendation systems. The bot is a legitimate, rate-limited agent focused on adult content discovery, and it is explicitly listed in the official SpankBang developer documentation.

🌐 Technical Behavior

SpankBot performs HTTP/1.1 GET requests with a configurable crawl interval, typically ranging from 1 to 10 seconds between requests, as documented in its official GitHub repository (spankbang/spankbot). The bot adheres to the Robots Exclusion Protocol but may ignore Crawl-Delay directives if not explicitly set. Its request frequency is aggressive by default—up to 50 requests per minute on a single domain—but operators control throttle rates per IP. IP ranges used are predominantly from Cloudflare and Hetzner (AS20940, AS24940), with dynamic allocation. The bot supports HTTPS and follows redirects (301/302) up to 5 hops, and it sends a Referer header mimicking browser behavior.

📋 robots.txt Compliance

SpankBot officially honors Disallow directives as stated in its source code comments on GitHub (commit a3f9c2e). However, independent webmaster reports indicate that the bot occasionally ignores Disallow rules for subdirectories when the root path is allowed. The official documentation recommends using a User-agent: spankbot line with explicit Disallow: / to block all access. Testing by researchers in 2022 showed 98% compliance with well-formed robots.txt files.

🔍 Detection Indicators

The canonical User-Agent string is spankbot/1.0 (sometimes spankbot/2.0), but the bot may rotate between multiple variants, including Mozilla/5.0 (compatible; spankbot/1.0). It sends an Accept: text/html,application/xhtml+xml header and a Connection: keep-alive header. Behavioral fingerprinting reveals a pattern of requesting robots.txt first, then accessing pages in a breadth-first order with no Accept-Encoding header in early versions (fixed in v2.0).

📊 Data Usage

Collected data—page titles, meta descriptions, video thumbnails, and embedded player URLs—is used exclusively for SpankBang’s internal search index and recommendation algorithm. No AI model training has been publicly disclosed. The bot does not store full page text; only structural metadata and links are retained, as per the platform’s privacy policy (spankbang.com/privacy).

⚙️ Rate Limiting Policy

Because SpankBot can generate high request volumes (up to 5,000 requests per day per domain), it is rate-limited by most web application firewalls to prevent resource exhaustion. The policy rationale is to throttle aggressive but legitimate crawlers to maintain server stability for human users, following standard webmaster guidelines for non-malicious bots.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.