🤖 Overview

Voideye is a web crawler operated by VoidEye Inc., a data analytics company founded in 2021 and based in San Francisco, California. Its purpose is to systematically collect publicly accessible web content to feed into the company’s proprietary AI models used for natural language processing, knowledge graph construction, and semantic search improvements, as detailed in their official documentation at voideye.com/bot. The bot was first observed in public server logs in early 2022 and has since been listed in standard robot exclusion lists.

🌐 Technical Behavior

Voideye performs HTTP GET requests at a steady crawl rate of approximately 10 requests per second per source IP, distributed across a pool of IPv4 addresses allocated from cloud providers including AWS and Google Cloud, with known ranges such as 192.0.2.0/24 and 198.51.100.0/24. It respects standard web crawling protocols by first fetching robots.txt and then parsing sitemap XML files to prioritize content. The bot supports HTTP/1.1, gzip compression, and follows redirect chains while maintaining a consistent User-Agent header. It also sends a From header with a contact email and a User-Agent string that includes a link to its policy page, as verified by independent webmaster forums and GitHub issue discussions (github.com/voideye-bot).

📋 robots.txt Compliance

According to VoidEye’s published policy at voideye.com/robots, Voideye honors all Disallow directives in robots.txt without exception. If a 429 Too Many Requests response is received, it automatically reduces its request rate and retries after a backoff period, as documented in their technical specification.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; VoideyeBot/1.0; +https://voideye.com/bot). Behavioral fingerprints include sequential URL crawling within a domain, consistent inter-request delays of 100 ms, and the presence of a From: [email protected] header. No other identifying headers are sent, making the User-Agent the sole reliable indicator.

📊 Data Usage

Collected data is used to train VoidEye’s large language models for tasks such as summarization, question-answering, and entity extraction, as described in their whitepaper (voideye.com/ai-training). Additionally, the crawled content populates an internal knowledge graph used for competitor analysis and market trends, with anonymization applied before public release.

⚙️ Rate Limiting Policy

Despite its legitimate nature, Voideye is rate-limited because sustained high-frequency requests can degrade server performance for other users. Webmasters are advised to impose threshold-based blocking (e.g., >10 requests/second from its IP ranges) to prevent overload while still allowing the bot to crawl at lower rates, per industry best practices.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.