xenu
Xenu Link Sleuth is a desktop application developed by Tilman Hausherr, first released in 1998 as a freeware tool for detecting broken links on websites. Unlike AI crawlers or search engine bots, it is an on‑demand link checker used by webmasters, SEO specialists, and content managers to locate dead links, redirect chains, and server errors. The tool downloads all pages from a given site to verify every hyperlink found, reporting HTTP status codes and anchor text.
Xenu honors robots.txt by default. When the crawler starts, it fetches the robots.txt file from the site root and obeys all Disallow directives. Users can disable this check by unchecking “Skip robots.txt check” in the settings. The behavior is documented in the built‑in help file and mirrored source code on GitHub. It does not respect Crawl‑Delay meta tags or HTTP headers unless manually configured in an advanced option file.
The primary User‑Agent string is Xenu Link Sleuth/1.3c (version may vary with releases). No additional custom headers like X‑Forwarded‑For are added. Behavioral fingerprints include sequential requests from a single IP address, absence of JavaScript or image loading, and a strong focus on scanning only href links without following redirects deeper than one hop. The tool also sends a Referer header set to the page where the link was found.
All collected data—broken links, redirect chains, page titles, anchor text, and HTTP response codes—are compiled into a local HTML report that the user can open in a browser. This report is not transmitted to any external server and is not used for AI training, advertising, or any third‑party service. The sole purpose is to aid website maintenance and quality assurance.
Although Xenu is not inherently aggressive, repeated scans or high concurrent thread counts can place significant load on a server. Web administrators often rate‑limit it by imposing per‑IP request ceilings (e.g., 5 requests per second) or using Web Application Firewall rules to throttle or temporarily block the tool when it exceeds reasonable thresholds. This policy protects site performance while still allowing legitimate link checking.
Similar Threats
Free Traffic Analysis
Discover which unwanted bots are being blocked on your site, how often they hit, and where they come from — real data from your own traffic, not guesswork.
🔍 Scan My Site FreePowered by JA4 fingerprinting, honeypot traps & behavioral analysis
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
Stay up to date with the latest from Boteraser.
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
These cookies are needed for adding comments on this website.
These cookies are used for managing login functionality on this website.
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
You can find more information in our Cookie Policy and Privacy Policy.