🤖 Overview

Pump is a legitimate web crawler operated by Pump Inc., a company specializing in AI‑powered search and content aggregation. First publicly documented in 2023, the Pump bot systematically indexes publicly accessible web pages to feed into the company’s proprietary search engine, which delivers structured, context‑aware results to end users.

🌐 Technical Behavior

The Pump crawler adheres to a polite crawl pattern, issuing requests at a rate of approximately 10–15 requests per second per IP, with a mandatory 500‑millisecond delay between successive requests to the same domain. It originates from a dynamic pool of IPv4 addresses owned by AWS (Amazon Web Services), primarily from the us‑east‑1 and eu‑west‑1 regions, as confirmed by the official Pump documentation at pump.co/crawler. The crawler uses HTTP/1.1 with a persistent connection header and sends a unique X‑Pump‑Crawler: true header in every request. It follows standard HTTPS redirects (HTTP 301/302) and respects canonical URLs via rel="canonical" link tags.

📋 robots.txt Compliance

Pump fully honours the Robots Exclusion Standard, as verified by its official documentation. It reads robots.txt at the root of every domain before each crawl session and honours both Disallow and Crawl‑delay directives. In independent tests conducted by webmasters, the bot was observed to enforce a 10‑second crawl delay when specified.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; PumpBot/1.0; +https://pump.co/bot). Secondary identifiers include the custom X‑Pump‑Crawler: true header and a reverse‑DNS entry of the form pump‑crawler‑*.pump.co. Log analysis confirms the bot always requests Accept: text/html,application/xhtml+xml and Accept‑Encoding: gzip.

📊 Data Usage

The collected data—including full page text, metadata, and structural elements—is used to train and refine Pump’s AI‑powered search ranking models and to populate its real‑time index. Pump Inc. states in its privacy policy that no personal or copyrighted content is stored beyond 30 days unless explicitly permitted by the site owner.

⚙️ Rate Limiting Policy

While Pump is a legitimate agent, its aggressive default crawl rate (up to 15 req/s) can impact server performance on shared hosting. Therefore, rate‑limiting to 5 requests per second per IP is recommended to maintain site stability without blocking the bot entirely.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.