🤖 Overview

mercator- is a web crawler originally developed by researchers at HP Labs and later adopted by Yahoo! for search indexing, as documented in the seminal 1999 paper “Mercator: A Scalable, Extensible Web Crawler” by Allan Heydon and Marc Najork (available via ACM Digital Library). It is operated today by Verizon Media (formerly Oath) as part of its web crawling infrastructure, primarily to feed data into Yahoo Search and related content analysis services. Unlike the more well-known “Yahoo! Slurp” bot, mercator- specifically refers to the user-agent string used by the production incarnation of this historical crawler, which continues to crawl a subset of the web for algorithmic improvements.

🌐 Technical Behavior

Mercator employs a breadth-first crawl strategy with a politeness delay enforced through a per-domain queue, as described in its original architecture. It sends standard HTTP GET requests with an Accept header of text/html,application/xhtml+xml and respects the Crawl-Delay directive in robots.txt. IP ranges are allocated from Verizon Media’s autonomous systems (e.g., AS 22394, AS 36408) and can be verified via reverse DNS lookups showing mercator-*.verizonmedia.net. Request frequency is typically moderate—around 1–2 requests per second per domain—but can spike during large-scale re-crawls. The crawler fetches only publicly accessible pages and does not attempt to index login-protected or client-side JavaScript-heavy content.

📋 robots.txt Compliance

Verizon Media officially states that all its crawlers, including mercator-, fully comply with the Robots Exclusion Protocol. Evidence from the official Yahoo Web Crawlers User-Agent Guide (available at developer.yahoo.com) confirms that it honors both Disallow and Crawl-Delay directives. Web server logs routinely show that mercator- stops crawling a site immediately after a correctly formatted Disallow: / rule is encountered.

🔍 Detection Indicators

The primary user-agent string is Mercator-1.0 or Mercator- followed by a version number; a common example seen in logs is Mercator-1.0 (Windows NT 6.1; rv:??) Gecko/20100101 Firefox/?? or simply Mercator-. Behavioral fingerprints include a consistent request pattern with an Accept-Language header set to en-US,en;q=0.5 and a lack of Referer header on initial requests. Some deployments also send a From header with a contact email like [email protected].

📊 Data Usage

Data collected by mercator- is used primarily for web indexing and ranking algorithms within Yahoo Search, as well as for academic research in distributed crawling and web graph analysis, per the original Mercator paper. The crawler also supports internal experiments on crawl efficiency and content freshness metrics at Verizon Media. No data is fed directly into third-party AI training sets, though aggregated web graph data may inform machine learning models used for spam detection and relevance scoring.

⚙️ Rate Limiting Policy

Because mercator- can generate a high volume of requests during re-crawl cycles (up to tens of thousands of pages per day per site), it is rate-limited using threshold-based blocking to prevent resource exhaustion. The rationale is that while the bot is legitimate, its aggressive schedule requires active throttling to maintain server stability without violating the crawler’s politeness policy.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.