selflinkchecker
selflinkchecker is a legitimate, lightweight web crawler developed and operated by an independent security researcher known as “selflinkchecker” for the purpose of detecting and reporting broken links, HTTP errors, and misconfigurations in public-facing websites. According to the official GitHub repository (github.com/selflinkchecker/selflinkchecker), the tool was first released in 2019 as an open-source link verification script written in Python, using the requests and BeautifulSoup libraries. Its primary goal is to help webmasters identify 404 errors, redirect chains, and internal linking issues, feeding results into a local report or online dashboard; it is not associated with any commercial search engine, AI training pipeline, or advertising network.
The selflinkchecker bot executes a sequential crawl of specified URLs, following only same‑domain links by default, with a configurable delay of 1 to 5 seconds between requests (typically 2 seconds). Based on the source code on GitHub, it uses a single user‑agent string and does not support concurrent threading by design. The crawler sends standard HTTP/HTTPS GET requests without cookies or JavaScript rendering, checking response status codes and timing out after 10 seconds. IP ranges are not fixed; the bot originates from the IP address of the machine running the check, which could be a home broadband or cloud VM. The tool respects the robots.txt by parsing it via the urllib.robotparser module, but the default configuration does not automatically enforce disallow rules — the operator must enable the “respect robots.txt” flag in the command line arguments. Known crawler cycles are short, typically finishing within minutes for small sites (under 1,000 pages).
According to the project’s documentation and the source code (github.com/selflinkchecker/selflinkchecker/blob/master/selflinkchecker.py), the tool includes a --respect-robots flag that, when set, causes the crawler to read the domain’s robots.txt and honor all Disallow directives. However, by default this flag is disabled because the author states many website owners want to check all URLs for broken links even if disallowed. For threat intelligence purposes, operators should assume the bot may ignore robots.txt unless explicitly configured otherwise. There are no documented cases of the bot being used for malicious scraping — it is intended as a diagnostic tool for the site owner.
The User-Agent string is selflinkchecker/1.0 (or selflinkchecker/2.0 in later versions) and is not randomized. The bot also sends an X-Requester header set to selflinkchecker as a self‑identification marker (source: GitHub issues). Behavioral fingerprints include a low request rate (average 0.5–1 request per second), no referrer header, and a consistent IP address for a given session. No other known User‑Agent variants exist.
The collected data is used solely by the website owner or the operator of the selflinkchecker instance to generate a list of broken links, status codes, and redirect chains on their own domain. No data is sent to a third‑party server unless the operator explicitly configures an export to a remote endpoint (e.g., via the --output-json flag). The project’s repository states that “no telemetry, analytics, or tracking is built into the tool.”
Because selflinkchecker is often run by site owners themselves on their own infrastructure, rate‑limiting is not typically required. However, if the bot appears from an unknown IP and ignores robots.txt, administrators may apply a conservative rate limit (e.g., 5 requests per minute per IP) to avoid impact on server resources. The rationale is that a misconfigured instance could accidentally escalate requests, though the tool’s default low speed makes it non‑aggressive.
⚠️
Unwanted bots inflate your analytics, drain server resources, and slow down real users. Check if your site is affected — completely free.
Check My Site for FreeFree to start · Cancel anytime
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
Stay up to date with the latest from Boteraser.
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
These cookies are needed for adding comments on this website.
These cookies are used for managing login functionality on this website.
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
You can find more information in our Cookie Policy and Privacy Policy.