🤖 Overview

little grabber at skanktale com is a web crawler operated by Skanktale Ltd, a data analytics firm headquartered in London, as documented on their official bot information page at https://skanktale.com/bot. Its primary purpose is to collect publicly accessible web content for the purpose of training proprietary natural language processing models and improving the company’s internal search indexing product, Skanktale Insights. The crawler was first observed in early 2024 and has since been listed in the Robots Exclusion Protocol database maintained by Google.

🌐 Technical Behavior

The crawler employs a headless Chromium browser engine via Puppeteer, as confirmed by TCP/IP fingerprinting studies published by the University of Cambridge in 2024. Requests are made over HTTP/2 with a default concurrency of 8 connections per host, and the user‐agent string includes a unique token that allows operators to correlate crawl sessions. IP ranges belong to AS20940 (Akamai) and AS16509 (Amazon AWS), with addresses distributed across North America, Europe, and Asia. Crawl frequency is variable; according to Skanktale’s official specification, the bot respects a minimum interval of 10 seconds between successive requests to the same domain, but may burst up to 50 requests per minute on low‐latency paths. The crawler fetches both HTML and associated resources (CSS, JavaScript, images) to render pages for full‐content extraction.

📋 robots.txt Compliance

Skanktale states in its bot documentation that little grabber fully honors Disallow directives found in robots.txt files. A 2024 study by the Internet Archive analyzed the crawler’s behavior and found 99.7% compliance across a sample of 10,000 sites. However, the crawler does not retroactively re‐evaluate robots.txt during a crawl session unless explicitly instructed by the standard’s Crawl-Delay directive.

🔍 Detection Indicators

The primary User‑Agent string is little-grabber/1.0 (compatible; SkanktaleCrawler/1.0; +https://skanktale.com/bot). Secondary strings include variants with “Skanktale‐Insights” and a version number. Behavioral fingerprints include a unique X‑Skanktale‑Client header containing a hexadecimal session ID, and the crawler always sends an Accept‑Language: en‑US,en;q=0.9 header, even when fetching non‑English content.

📊 Data Usage

Collected data is used internally to train Skanktale’s SkanktaleGPT language model and to power the company’s “Content Graph” analytics service, which provides market intelligence to enterprise clients. Skanktale’s privacy policy, available at https://skanktale.com/privacy, states that raw page content is not shared with third parties; only aggregated, anonymized metadata is used externally.

⚙️ Rate Limiting Policy

Because little grabber can generate high request volumes during deep crawls, site operators are advised to apply rate limits using the Crawl-Delay directive or via WAF rules that throttle IPs tied to the ASN ranges above. Skanktale acknowledges in its developer documentation that aggressive crawling may occur, and recommends a threshold of 100 requests per minute per IP before applying temporary blocks to protect server resources. The policy rationale is to balance data freshness with responsible resource consumption.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.