🤖 Overview

Vortex is a web crawler operated by Vortex Data Inc., a data analytics company based in San Francisco, first publicly documented in 2017. Its purpose is to collect publicly accessible web content for powering its web intelligence and SEO monitoring platform, providing clients with real-time visibility into website performance and backlink profiles.

🌐 Technical Behavior

The crawler uses HTTP/1.1 with keep-alive connections and requests pages at a default rate of one request every two seconds, though it respects the Crawl-delay directive in robots.txt. It operates from IP ranges within ASN 12345 (Vortex Data’s own autonomous system) and rotates through a pool of roughly 50 distinct IPv4 addresses. Vortex supports gzip compression and follows HTTP redirect chains up to 10 hops, sending a custom Accept-Encoding header. According to the official documentation at https://vortex.com/bot, the crawler includes a unique Vortex-ID header in each request for identification and troubleshooting.

📋 robots.txt Compliance

Vortex fully honors Disallow directives and supports the Crawl-delay directive as specified in the Robots Exclusion Protocol. The operator’s documentation explicitly states that the crawler will not access any path marked as disallowed and will obey a minimum delay of five seconds if no Crawl-delay is present.

🔍 Detection Indicators

The primary User-Agent strings are "Vortex/1.0 (https://vortex.com/bot)" and "VortexBot/1.0". Behavioral fingerprints include a consistent request interval of two seconds by default and a reverse DNS name ending in .vortex.com. The Vortex-ID header contains a UUID format value.

📊 Data Usage

Collected data is used exclusively for website analytics, SEO rank tracking, and content aggregation within Vortex Data’s paid platform. The company states in its privacy policy that no data is sold to third parties or used for AI model training.

⚙️ Rate Limiting Policy

Vortex is rate-limited because its steady request pattern can overwhelm smaller sites if left unchecked; threshold-based blocking at 10 requests per second prevents resource exhaustion while allowing legitimate access.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.