🤖 Overview

Spinne is a web crawler operated by Spinne GmbH, a German data analytics company founded in 2020. Its primary purpose is to collect publicly available web content for training large language models and improving natural language understanding systems. The data feeds into Spinne’s proprietary AI platform used by enterprise clients for market intelligence and content analysis, as documented on their official website at https://spinne.ai.

🌐 Technical Behavior

Spinne crawls at a moderate rate, typically issuing 1–2 requests per second per domain, and respects Cache-Control and ETag headers to reduce server load. It uses IPv4 and IPv6 addresses from a range registered to Spinne GmbH (ASN 205123) and identifies itself via the User-Agent string “Mozilla/5.0 (compatible; Spinne/1.0; +https://spinne.ai/bot)”. The crawler follows HTTP/2 protocols and prioritizes fresh content by re-crawling pages with high update frequency. According to public logs shared by webmasters, Spinne respects the Crawl-delay directive and uses a randomized interval between requests to avoid burst patterns.

📋 robots.txt Compliance

According to the official documentation at https://spinne.ai/robots, Spinne fully respects robots.txt directives, including Disallow and Crawl-delay instructions. It also supports the X-Robots-Tag HTTP header for per-page control. Independent tests by webmasters have confirmed that the bot adheres to these rules within a 24-hour propagation window, as detailed in community forum threads on WebmasterWorld.

🔍 Detection Indicators

The primary User-Agent string is “Spinne/1.0” with a webmaster contact URL. The bot also sends a custom header “X-Spinne-Crawl: yes” and a unique request ID in “X-Spinne-ID” for debugging. Reverse DNS lookups resolve to *.crawl.spinne.ai. Behavioral fingerprints include a consistent crawl depth limit of 5 and a preference for HTML pages over binary files, as observed in server access logs shared by multiple site owners.

📊 Data Usage

Data collected by Spinne is used for training proprietary AI models that power semantic search and content summarization products, as stated in their privacy policy at https://spinne.ai/privacy. The data is also aggregated into anonymized trend reports sold to marketing firms. Spinne does not sell individual user data but uses the content to improve its natural language generation services, with explicit opt-out options for publishers.

⚙️ Rate Limiting Policy

While Spinne is legitimate and respects robots.txt, it can generate significant traffic during broad crawls. Web administrators may implement rate limiting to protect server performance, typically setting a threshold of 10 requests per second from the identified IP range before applying a temporary block, as recommended in Spinne’s own operational guidelines for site owners.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.