🤖 Overview

LinkSweeper is a legitimate automated security crawler operated by Microsoft as part of Microsoft Defender for Office 365 (formerly Office 365 Advanced Threat Protection). Its purpose is to proactively scan URLs embedded in emails, documents, and messages to detect phishing, malware, and other malicious content, protecting users before they click.

🌐 Technical Behavior

LinkSweeper performs real-time URL crawling and re-crawling at intervals to verify the safety of previously benign links. It uses HTTP/HTTPS requests with a typical frequency of multiple requests per second per URL, though overall crawl volume is distributed across Microsoft's global infrastructure. The crawler originates from Microsoft Azure and Office 365 IP ranges published in the official Microsoft IP Ranges and Service Tags JSON. It may also follow redirects and inspect page content, including certificates and domain registration data, to assess threat levels. The bot does not execute JavaScript but may fetch HTML and headers.

📋 robots.txt Compliance

According to Microsoft's own documentation for its crawlers, LinkSweeper respects the robots.txt protocol when crawling public websites. It checks for Disallow directives and will not crawl paths that are explicitly forbidden. However, because the service is security-focused, some administrators report that LinkSweeper may still access certain URLs if they are embedded in trusted emails, even if disallowed — but Microsoft officially states compliance with standard crawl rules.

🔍 Detection Indicators

The primary User-Agent string for LinkSweeper is "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; LinkSweeper)". Additionally, requests often include a X-Forwarded-For header or originate from known Microsoft IP ranges. Behaviorally, the bot requests only the first few kilobytes of a page and does not download large assets. It may also include a Referer header matching the email or document source.

📊 Data Usage

The data collected by LinkSweeper — page content, headers, SSL certificates, and redirect chains — is used exclusively for security analysis. Microsoft processes this data to update its threat intelligence feed, block malicious URLs in real-time, and generate alerts for Office 365 admins. No data is used for AI training or search indexing; the sole purpose is protective.

⚙️ Rate Limiting Policy

Because LinkSweeper can generate high volumes of requests during outbreak events, it is rate-limited by web administrators to prevent performance degradation. Threshold-based blocking is recommended, allowing legitimate security scanning while throttling excessive burst traffic, with a typical policy of allowing up to 10 requests per second per IP and blocking if exceeded.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.