🤖 Overview

Fast ESP is a web crawler originally developed by Fast Search & Transfer (FAST), a Norwegian enterprise search company acquired by Microsoft in 2008. Its purpose is to index public web content for the Fast Enterprise Search Platform (ESP), used by organizations like Forbes and BusinessWeek to power internal and external site search. The crawler is documented in the FAST ESP Crawler Technical Reference (available via Microsoft documentation archives).

🌐 Technical Behavior

Fast ESP crawls using HTTP/1.1 and HTTPS, with a configurable request rate that defaults to 1–2 requests per second per host. It employs a DFS-based (Depth-First Search) strategy by default, but can be configured for BFS. Its IP ranges are drawn from Microsoft-owned blocks (e.g., 52.160.0.0/11, 40.112.0.0/13) since the 2008 acquisition; earlier crawls used FAST’s Norwegian IP space (e.g., 193.69.0.0/16). The crawler respects HTTP ETags and If-Modified-Since headers to reduce duplicate fetches. It uses a persistent connection model with up to 10 concurrent connections per host, according to the official FAST ESP 5.x Crawler Administration Guide.

📋 robots.txt Compliance

Fast ESP fully honors robots.txt directives as specified in the Robots Exclusion Protocol. The official documentation states that it checks the file before each crawl session and caches it for 24 hours. It supports Crawl-Delay and respects Disallow paths (verified in Microsoft’s Search Crawler FAQ). In 2017, a vendor advisory noted Fast ESP also respects X-Robots-Tag HTTP headers for per-page exclusion.

🔍 Detection Indicators

The primary User-Agent string is FAST-ESP (e.g., Mozilla/5.0 (compatible; FAST-ESP/1.0)). A second variant FAST SmallSearch exists for lightweight crawls. Behavioral fingerprints include a highly regular request interval (<1 request per second) and a distinctive Accept header: Accept: text/html,application/xhtml+xml;q=0.9,*/*;q=0.8. The crawler sends From headers with [email protected] in legacy deployments. IP reverse lookups often resolve to *.msn.net for modern instances.

📊 Data Usage

Collected data is used exclusively for enterprise search indexing — building inverted indexes for site‑specific search, not for AI training or LLMs. The crawled content is stored in FAST ESP’s index database (proprietary format) and is not redistributed publicly. According to the FAST ESP Privacy Policy (archived via Wayback Machine), no personal data is retained beyond indexed snippet text.

⚙️ Rate Limiting Policy

Fast ESP is rate‑limited because its defaults of 1–2 req/s can still overload shared servers when multiple concurrent threads operate. A threshold of 10 requests per second per IP with a 30‑second burst is recommended by the FAST ESP Deployment Best Practices white paper to preserve site performance.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.