🤖 Overview

PageDown is a web crawler operated by PageDown Inc., a company specializing in web content extraction for AI training datasets. The bot was first documented in a 2023 whitepaper titled "Large-Scale Web Crawling for Language Model Pre-training" (available at pagedown.com/research). Its purpose is to collect publicly available text from diverse web sources to feed into the company's proprietary language model training pipeline.

🌐 Technical Behavior

PageDown crawls with an average request rate of 2 requests per second per domain, with bursts up to 5 req/s during initial index phases. It uses HTTP/1.1 and HTTP/2 protocols and respects the Accept-Encoding header for gzip and brotli. The crawler originates from IP ranges registered under ASN 12345 (PageDown Networks), which are published in a dedicated PTR record file at pagedown.com/ips.txt. Crawl depth is limited to 20 hops by default, and it avoids duplicate content via ETag and Last-Modified headers.

📋 robots.txt Compliance

Based on the official documentation at pagedown.com/bot-policy, PageDown fully respects robots.txt directives. It checks for Disallow rules before fetching each URL and also honors Crawl-Delay directives. A fallback delay of 5 seconds is applied if no Crawl-Delay is specified.

🔍 Detection Indicators

The primary User-Agent string is PageDownBot/1.0 (+https://pagedown.com/bot). Additional identifiers include the X-PageDown-Bot header set to true. The bot's requests typically include a From header with an email contact, e.g., [email protected]. The User-Agent is also present in the Via header when behind a proxy.

📊 Data Usage

Collected data is used exclusively for training PageDown's AI language models, as detailed in their privacy policy (pagedown.com/privacy). The company does not resell data and applies content filtering to remove personally identifiable information. Crawled pages are stored in a distributed database and used to generate training corpora for natural language understanding tasks.

⚙️ Rate Limiting Policy

Although PageDown is a legitimate bot, its high-volume concurrent requests can impact server performance if unregulated. Rate limiting is applied to prevent resource exhaustion — typical thresholds block if a single IP exceeds 10 req/s over a 30-second window — while still allowing the bot to complete its indexing within a reasonable timeframe.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.