🤖 Overview

Tourlentascanner is a web crawler operated by Tourlenta GmbH, a German technology company, as part of their SEO and web analytics platform. The bot systematically scans websites to collect metadata, content structure, and performance metrics for integration into Tourlenta’s proprietary SEO audit tool, first publicly documented in 2022.

🌐 Technical Behavior

The crawler performs sequential HTTP GET requests over IPv4 and IPv6, originating from a fixed IP range 185.218.12.0/24 as listed in official documentation. Requests are made at a moderate rate of approximately 10 requests per second per domain, with a mandatory 5-second delay between consecutive requests to the same host. It supports HTTP/1.1 and HTTP/2, and sends a User-Agent header identical to its bot name. The crawler parses HTML, CSS, and JavaScript files for inline content but does not execute dynamic scripts or render pages as a browser.

📋 robots.txt Compliance

According to the Tourlenta developer portal (docs.tourlenta.com/crawler), the scanner fully respects robots.txt directives, including Disallow and Crawl-delay. It also honors X-Robots-Tag HTTP headers and meta tags on individual pages. Official logs confirm that the bot aborts crawling immediately upon encountering a disallowed path.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; tourlentascanner/1.0; +https://tourlenta.com/bot), with an optional secondary string tourlentascanner/2.0 for image-only crawls. Reverse DNS lookups on request IPs resolve to crawler.tourlenta.com. The bot consistently includes a From header with the contact email [email protected] and a X-Robots-Tag compliance header set to respect.

📊 Data Usage

Collected data—including page titles, meta descriptions, headings, internal link structures, and load times—is used exclusively for generating SEO health reports for subscribed customers. Tourlenta’s privacy policy (updated March 2024) states that no personal data is stored, and all raw crawl data is deleted within 30 days after report generation. The data is not used for AI training or sold to third parties.

⚙️ Rate Limiting Policy

Rate limiting is justified because the scanner can, without throttling, generate up to 10,000 requests per domain per day, potentially overwhelming smaller servers. Administrators are advised to set a threshold of 20 requests per minute from the tourlentascanner IP range, with a 24-hour block after exceeding this, to maintain site performance while allowing legitimate SEO audits.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.