xenu

Bot User-Agent: xenu

🤖 Overview

Xenu Link Sleuth is a desktop application developed by Tilman Hausherr, first released in 1998 as a freeware tool for detecting broken links on websites. Unlike AI crawlers or search engine bots, it is an on‑demand link checker used by webmasters, SEO specialists, and content managers to locate dead links, redirect chains, and server errors. The tool downloads all pages from a given site to verify every hyperlink found, reporting HTTP status codes and anchor text.

🌐 Technical Behavior

Xenu performs a recursive crawl starting from a seed URL, following all href links it encounters. By default, it operates in single‑threaded mode but can be configured to use up to 100 concurrent threads via the “Options” dialog. Each HTTP GET request has a configurable timeout (default 30 seconds). The tool respects server‑side delays and supports HTTPS, cookies, basic authentication, and form‑based logins. Crawl speed is user‑dependent; common rates range from 1–5 requests per second, though high concurrency settings may exceed 50 requests per second. Xenu does not maintain fixed IP ranges—it uses the IP address of the machine on which it runs. It also checks FTP and email links (mailto:) for validity, though email verification is limited to syntax checks.

📋 robots.txt Compliance

Xenu honors robots.txt by default. When the crawler starts, it fetches the robots.txt file from the site root and obeys all Disallow directives. Users can disable this check by unchecking “Skip robots.txt check” in the settings. The behavior is documented in the built‑in help file and mirrored source code on GitHub. It does not respect Crawl‑Delay meta tags or HTTP headers unless manually configured in an advanced option file.

🔍 Detection Indicators

The primary User‑Agent string is Xenu Link Sleuth/1.3c (version may vary with releases). No additional custom headers like X‑Forwarded‑For are added. Behavioral fingerprints include sequential requests from a single IP address, absence of JavaScript or image loading, and a strong focus on scanning only href links without following redirects deeper than one hop. The tool also sends a Referer header set to the page where the link was found.

📊 Data Usage

All collected data—broken links, redirect chains, page titles, anchor text, and HTTP response codes—are compiled into a local HTML report that the user can open in a browser. This report is not transmitted to any external server and is not used for AI training, advertising, or any third‑party service. The sole purpose is to aid website maintenance and quality assurance.

⚙️ Rate Limiting Policy

Although Xenu is not inherently aggressive, repeated scans or high concurrent thread counts can place significant load on a server. Web administrators often rate‑limit it by imposing per‑IP request ceilings (e.g., 5 requests per second) or using Web Application Firewall rules to throttle or temporarily block the tool when it exceeds reasonable thresholds. This policy protects site performance while still allowing legitimate link checking.

Free Bot Analysis

Is Your Site Under Bot Attack Right Now?

Find out exactly how much of your traffic is automated — and which bots are draining your bandwidth and skewing your analytics.

Run Free Bot Scan →

No credit card required  ·  Results in minutes

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.