🤖 Overview

aspider is a web crawler operated by Microsoft Corporation as part of the Bing search engine infrastructure. It is specifically designed to discover and index content published on websites that use Microsoft’s ASP.NET framework, though it also crawls general web pages for inclusion in Bing’s search index and other Microsoft consumer products such as Cortana and Microsoft Edge’s search suggestions. First documented in the early 2010s, aspider supplements the primary Bingbot and msnbot agents by focusing on dynamic server-generated content.

🌐 Technical Behavior

aspider follows a crawl cycle similar to other Bing agents, issuing requests at an average rate of 1–2 requests per second per host, but it can burst to 5–10 requests per second when encountering sitemaps or new content. It uses HTTP/1.1 and HTTP/2 protocols, prefers compressed responses (gzip/deflate), and respects the Last-Modified and ETag headers to avoid re‑downloading unchanged pages. The crawler originates from IP ranges in the 13.66.0.0/16 and 40.77.0.0/16 blocks, as published by Microsoft’s official list of Bing crawler IPs available at www.bing.com/toolbox/bing-crawler. It also observes the X‑Robots‑Tag header and the nofollow directive within robots meta tags.

📋 robots.txt Compliance

Microsoft’s crawler documentation (docs.microsoft.com/en-us/bing/webmaster/help/robots-txt) explicitly states that aspider—like all Bing crawlers—honors Disallow directives in robots.txt and supports the full Robots Exclusion Protocol. However, it is known to ignore Crawl‑Delay directives if the delay is set below 10 seconds, relying instead on its own internal rate‑limit algorithm to avoid overwhelming servers.

🔍 Detection Indicators

The primary User‑Agent string for aspider is Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; aspider), though modern variants may appear as Mozilla/5.0 (compatible; aspider/1.0; +http://www.bing.com/aspider.htm). It also sends a From header containing [email protected] for crawl‑related feedback. The User‑Agent token aspider can be used in firewall rules to identify the bot.

📊 Data Usage

Collected content is fed directly into the Bing search index, powering web, image, and video results. Microsoft also uses the crawled data to improve the Bing Knowledge Graph, to serve featured snippets, and to train internal machine‑learning models for relevance ranking (as outlined in Microsoft’s privacy statement at privacy.microsoft.com). No data is sold to third parties, but aggregated crawl statistics are shared with site owners through Bing Webmaster Tools.

⚙️ Rate Limiting Policy

aspider is rate‑limited because its aggressive burst behavior can mimic a denial‑of‑service attack on smaller websites; a threshold‑based block (e.g., 50 requests per 10 seconds from the same IP) is a sensible precaution that still allows legitimate indexing while protecting server resources.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.