🤖 Overview

RankFlex is a legitimate web crawler operated by RankFlex Inc., first documented in early 2024, designed to collect publicly accessible web content for training large language models (LLMs) and improving search indexing algorithms. The data feeds into RankFlex's proprietary AI platform used for natural language processing and semantic search. According to the company's official bot policy page at https://rankflex.com/bot, the crawler only targets openly available content and does not bypass authentication or paywalls.

🌐 Technical Behavior

RankFlex employs a distributed crawl architecture using IP addresses primarily within the 205.185.115.0/24 range, as documented in security advisories from Cloudflare and Akamai. Requests are sent over HTTP/1.1 with a default crawl delay of approximately 2 seconds between requests per IP, which can be adjusted via the Crawl-Delay directive in robots.txt. The crawler does not execute JavaScript or fetch embedded resources; it only retrieves static HTML. RankFlex respects the X-Robots-Tag HTTP header and uses a configurable throttle that slows down if the server returns frequent errors or high response times.

📋 robots.txt Compliance

RankFlex fully honors robots.txt Disallow and Allow directives, as confirmed by its official documentation. The crawler fetches the robots.txt file at the start of each session and caches it for up to 24 hours. No known incidents of non‑compliance have been reported in public security databases or community forums.

🔍 Detection Indicators

The primary User‑Agent string is RankFlex/1.0 (compatible; RankFlexBot; +https://rankflex.com/bot). A secondary Mozilla‑based variant is also used: Mozilla/5.0 (compatible; RankFlex/1.0; +https://rankflex.com/bot). Additionally, every request includes a custom X‑Robot‑Name: RankFlex header. Behavioral fingerprints include a fixed inter‑request interval and a lack of browser‑like headers such as Accept‑Language.

📊 Data Usage

Collected data is exclusively used for training RankFlex's large language models and improving their semantic search index. According to their privacy policy, data is not shared with third parties, used for advertising, or employed for user profiling. RankFlex publishes a quarterly transparency report detailing crawl volumes and domain coverage.

⚙️ Rate Limiting Policy

Although RankFlex is a benign crawler, its initial crawl of a large site can produce significant request volumes. Rate limiting via a threshold‑based policy (e.g., blocking IPs that exceed 100 requests per minute) is recommended per OWASP guidelines to prevent resource exhaustion while still allowing legitimate access.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.