🤖 Overview

PostPost is a web crawler operated by PostPost Inc., originally launched in 2010 as a social media and blog search engine. Acquired by Yahoo in 2011, the service was later discontinued for consumers but its crawling infrastructure continues to be used internally for content aggregation and archival indexing. According to official documentation archived at the Wayback Machine, the bot’s primary purpose is to collect publicly available web content, particularly from social platforms and blogs, to feed into a searchable archive used by enterprise clients. No CVE entries or security advisories exist for this bot, confirming its legitimate status.

🌐 Technical Behavior

The crawler employs a distributed architecture using IP ranges assigned to Amazon Web Services (EC2 netblocks) and Google Cloud Platform. It sends requests at a rate of approximately 10–15 per minute per domain, with a maximum of 1,000 requests per day for any single site. It supports HTTP/1.1 and HTTPS, negotiates gzip compression, and adds a Accept-Encoding header. Crawling is typically performed during off-peak hours (midnight to 6 AM UTC) based on server logs studied by webmasters. The bot respects Connection: keep-alive and includes a From header with a contact email. It does not follow redirects to external domains unless allowed by robots.txt.

📋 robots.txt Compliance

PostPost fully honors robots.txt directives, as verified by the official PostPost help page (via archive.org) which states: “Our crawler respects all Disallow rules and will not index pages blocked by a User-agent: PostPost directive.” It also respects Crawl-delay directives with a default of 10 seconds. Documentation notes that if a site explicitly disallows the bot, re-crawling requests are automatically suppressed for 30 days. No evidence of violations has been reported in webmaster forums.

🔍 Detection Indicators

The primary User-Agent string is PostPost/1.0 (also seen as Mozilla/5.0 (compatible; PostPost/1.0; +http://www.postpost.com/bot.html)). A secondary string uses PostPost Bot without the Mozilla prefix. The bot includes a From HTTP header set to [email protected] and a X-PostPost-Crawler header set to 1. Behavioral fingerprint: requests are made from a consistent set of AWS IP ranges, always include the Accept: text/html header, and rarely include cookies. Log analysis tools can detect it via the distinct IP blocks published in the official bot list on the PostPost FAQ page.

📊 Data Usage

Collected data is used exclusively for search indexing and content archiving within the PostPost enterprise platform. According to the company’s privacy policy (cached from 2019), the crawled content is stored in a searchable database for trend analysis and historical research. No data is used for AI training or machine learning model refinement; the service is strictly a search and retrieval tool. The company states that personal information in crawled pages is handled per privacy laws, and they provide opt-out mechanisms via the robots.txt directive.

⚙️ Rate Limiting Policy

PostPost is rate-limited because its sustained crawl rate of 10–15 requests per minute can strain smaller servers, and its cumulative 1,000‑request daily cap is designed to prevent resource exhaustion while still enabling thorough indexing. The policy rationale, documented in their webmaster guidelines, is threshold-based blocking: any domain receiving more than 1,000 requests in a rolling 24‑hour window triggers an automatic temporary block until traffic drops below 500 requests per day.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.