🤖 Overview

RankurBot is a web crawler operated by Rankur, a SEO analytics company based in San Francisco, launched in 2019. Its purpose is to gather publicly accessible web content for Rankur's keyword rank tracking, backlink analysis, and competitive intelligence platform.

🌐 Technical Behavior

The bot sends HTTP GET requests with standard Accept headers for text/html and application/xhtml+xml, using HTTP/1.1 with gzip compression. It respects If-Modified-Since headers to reduce bandwidth usage. Crawl intervals default to 3 seconds per domain. IP addresses originate from AWS (us-east-1, eu-west-1) and DigitalOcean (nyc1, ams3) with reverse DNS like crawler-01.rankur.io. Maximum crawl depth is 10 levels, and it follows Canonical tags and redirects up to 5 hops.

📋 robots.txt Compliance

RankurBot fully adheres to robots.txt Disallow directives as documented at https://rankur.com/bot-policy. It also obeys Crawl-Delay directives, setting a minimum delay of 2 seconds when specified. The bot checks robots.txt at the start of each crawl session and caches it for up to 1 hour.

🔍 Detection Indicators

Definitive User-Agent: Mozilla/5.0 (compatible; RankurBot/1.0; +https://rankur.com/bot). Custom HTTP header X-Rankur-Bot set to true. Typical IP ranges: 15.197.0.0/16 (AWS), 138.197.0.0/16 (DigitalOcean). The bot does not execute JavaScript and requests only static content.

📊 Data Usage

Collected data feeds Rankur's SEO dashboard for keyword tracking, backlink discovery, and competitor analysis. Stored data includes page titles, meta descriptions, H1 tags, and link graphs. Data is retained for 90 days then aggregated; it is not sold nor used for AI training.

⚙️ Rate Limiting Policy

RankurBot can be aggressive when monitoring many client sites simultaneously. Rate limiting at 20 requests per minute per IP preserves server resources while allowing legitimate SEO data collection. Webmasters can contact Rankur support for custom crawl schedules.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.