inktomi
Bot User-Agent:inktomi
🤖 Overview
Inktomi was originally developed by Inktomi Corporation (founded 1996) as the web-crawling engine behind its own search service. After Yahoo! acquired Inktomi in 2003, the crawler was rebranded as Yahoo! Slurp and continues to be used for Yahoo Search indexing. Its primary purpose is to discover and fetch publicly accessible web content for inclusion in Yahoo’s search results.
🌐 Technical Behavior
The Inktomi/Slurp crawler operates as a distributed, multi-threaded system that sends sequential HTTP or HTTPS GET requests. Historically, it has been observed to request multiple pages from the same domain in rapid succession, often exceeding 10 requests per second under normal conditions. The crawler uses a rotating set of IP addresses belonging to Yahoo’s owned blocks (e.g., 98.137.0.0/16 and 74.6.0.0/16), though ranges have changed over time. It respects the Crawl-Delay directive in robots.txt but does not guarantee a fixed minimum delay. The crawler also follows noindex meta tags and X-Robots-Tag headers as specified in the robots exclusion protocol.
📋 robots.txt Compliance
Yahoo! Slurp is documented to honor Disallow directives in robots.txt, as verified by Yahoo’s own webmaster guidelines (https://help.yahoo.com/kb/search/slurp-crawler-ip-addresses). However, historical reports indicate occasional failures to fully comply with specific patterns, though these are rare. The crawler supports Crawl-Delay and Allow directives, making it one of the more compliant large-scale bots.
🔍 Detection Indicators
The most reliable detection method is the User-Agent string: typically Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) or the older Inktomi Slurp. Some versions also include Slurp.so/1.0 or Slurp/2.0. The crawler may not always present a consistent From header. Behavioral fingerprints include high request frequency (often >1/second), repetitive crawling of the same URL with no query string variation, and an absence of JavaScript or cookie support.
📊 Data Usage
The collected content is strictly used for building and updating Yahoo’s web search index. Historical materials (e.g., Wikipedia article on Inktomi) confirm that the data is not repurposed for AI model training, as the crawler predates the generative AI era. Yahoo may also use crawled pages for snippet generation and ranking signals, but does not sell the raw data or use it for advertising targeting.
⚙️ Rate Limiting Policy
Because Inktomi/Slurp can issue a high volume of requests in a short time—often exceeding typical user traffic—webmasters are advised to rate-limit it at the server or CDN level. A threshold of 10–20 requests per second per IP is commonly recommended, with a 429 response for exceedances, to prevent resource exhaustion while still allowing the crawler to index content.
Similar Threats
🛡️
Stop Bots. Save Bandwidth. Protect Revenue.
Boteraser automatically detects and blocks unwanted bots — protecting your site from scrapers, DDoS bursts, and credential stuffing attacks without slowing down real visitors.
✅ Start Free ProtectionSetup takes under a minute · Free trial available
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
