🤖 Overview

RankurBot is a legitimate web crawler operated by Rankur, a search engine and web analytics platform headquartered in Singapore. First documented in 2019, its primary purpose is to index publicly accessible web pages for the Rankur search engine, which provides users with relevant search results and site-ranking data. According to the official Rankur documentation (rankur.com/crawler), the bot collects metadata, content summaries, and link structures to feed the Rankur index and associated analytics dashboards. It is not associated with any malicious activity and is operated under strict data-handling policies.

🌐 Technical Behavior

RankurBot employs a breadth-first crawl strategy with a default request frequency of 2 requests per second per domain, as noted in the official robot.txt guidelines. It uses an IPv4 range from 103.235.46.0/24 (assigned to Rankur's AWS infrastructure) and an IPv6 range from 2a05:d014::/29. The bot sends standard HTTP GET requests with a modified Accept-Encoding: gzip, deflate header and does not support cookies. It respects the If-Modified-Since header for bandwidth efficiency. Crawling occurs 24/7 but is throttled during peak hours (08:00–20:00 UTC) via a token-bucket algorithm, as described in Rankur's engineering blog (rankur.com/engineering/crawl-patterns). The bot avoids JavaScript-rendered content unless explicitly allowed via Allow: /?dynamic directives. It also identifies itself via a From header with a contact email ([email protected]).

📋 robots.txt Compliance

RankurBot strictly honors Disallow and Allow directives in robots.txt, as confirmed by multiple third-party tests (e.g., Moz's crawler tester in 2021). The official Rankur documentation states that it will not crawl URLs with a Disallow directive, and it supports pattern-based rules (e.g., Disallow: /private/). However, it does not honor Crawl-Delay directives because its rate limiting is server-side; site owners must instead contact Rankur to adjust crawl intervals. Non-compliance is rare and typically due to misconfigured robots.txt syntax.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; RankurBot/1.0; +https://rankur.com/crawler). A secondary string for mobile content is Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.91 Mobile Safari/537.36 (compatible; RankurBot-Mobile/1.0). Behavioral fingerprints include requests without Accept-Language headers and a consistent request interval pattern of exactly 500 milliseconds between requests to the same domain. Server logs show a distinctive X-Rankur-Bot header set to true on all requests.

📊 Data Usage

Collected data is used exclusively for search indexing and aggregate analytics on Rankur’s platform. According to Rankur’s privacy policy (rankur.com/privacy), text content, link structures, and metadata are stored for up to 90 days in temporary caches before being stripped of personally identifiable information (PII). The data feeds the Rankur search ranking algorithm and public site statistics pages. It is not used for AI training or advertising profiling. A small subset (less than 1% of crawled pages) may be retained for quality assurance and debugging for up to 18 months.

⚙️ Rate Limiting Policy

RankurBot is rate-limited because its crawl frequency can overwhelm poorly optimized servers, especially shared hosting environments. The policy rationale for threshold-based blocking (e.g., 2 requests/sec per domain) is to allow site owners to protect server resources while still benefiting from Rankur’s indexing. If a domain consistently exceeds the threshold, RankurBot automatically pauses crawling for that domain for 24 hours and notifies the webmaster via the provided contact email. This approach balances crawl efficiency with server load management.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.