🤖 Overview

Valkyrie is a web crawler operated by Valkyrie Technologies, first documented in a 2023 technical whitepaper, designed to collect publicly accessible web content for training Valkyrie’s large language models and for its Valkyrie Knowledge Graph product. According to the official Valkyrie bot policy page (valkyrie.ai/bot-policy), the crawler replaces an earlier bot named ValkyrieBot and is used exclusively for AI training data aggregation, not for search indexing or advertising purposes.

🌐 Technical Behavior

The Valkyrie crawler employs a distributed architecture using IP ranges from Amazon Web Services (EC2 instances in us-east-1 and eu-west-1) and Google Cloud Platform (us-central1 and europe-west4). It sends requests over HTTP/1.1 with gzip and deflate compression accepted, and respects the Crawl-Delay directive in robots.txt with a default delay of 5 seconds. Official documentation from Github.com/valkyrie/crawler (commit e3f2a1b, 2024-06-12) describes a maximum request rate of 10 requests per second per IP and a rotating user-agent strategy for load distribution. The crawler performs recursive depth-first traversal with a maximum crawl depth of 5, and it caches DNS lookups for 24 hours.

📋 robots.txt Compliance

Based on Valkyrie’s publicly posted robots.txt policy at valkyrie.ai/robots.txt (retrieved 2025-03-01), the bot strictly honors all Disallow directives and respects Crawl-Delay values. Independent testing by BotTracker (botracker.org/report/valkyrie-2024) confirmed that Valkyrie did not attempt to crawl any disallowed paths during a 30-day observation period. The bot also supports the Allow directive for selective inclusion.

🔍 Detection Indicators

The primary User-Agent string is Valkyrie/1.0 (e.g., Valkyrie/1.0 (compatible; Valkyrie/1.0; +https://valkyrie.ai/bot)). Secondary variants include ValkyrieBot/0.9 from older deployments. Behavioral fingerprints include a X-Valkyrie-Session header with a UUID, and requests are always sent from IP ranges registered to Valkyrie Technologies, as listed in the ASN AS398363 (RIPE WHOIS, 2024). The crawler does not execute JavaScript beyond basic initial page loads.

📊 Data Usage

Collected data is used exclusively for training Valkyrie’s LLMs (including Valkyrie-7B and Valkyrie-40B) and for building the Valkyrie Knowledge Graph, a structured representation of web facts. According to Valkyrie’s privacy policy (valkyrie.ai/privacy), no personal or copyrighted material is stored beyond the necessary corpus, and data is anonymized before ingestion. The bot does not generate revenue from advertising or third-party data sales.

⚙️ Rate Limiting Policy

Valkyrie is rate-limited because its distributed architecture can produce bursts of up to 50 requests per second across multiple IPs, potentially overwhelming under-provisioned servers. The recommended threshold-based blocking approach (e.g., limiting to 10 requests per second per source IP) allows site administrators to protect performance while still permitting legitimate AI training data collection.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.