Skip to main content

Boteraser | Website and Server Security Solutions

selflinkchecker

Monitor User-Agent: selflinkchecker

🤖 Overview

selflinkchecker is a legitimate, lightweight web crawler developed and operated by an independent security researcher known as “selflinkchecker” for the purpose of detecting and reporting broken links, HTTP errors, and misconfigurations in public-facing websites. According to the official GitHub repository (github.com/selflinkchecker/selflinkchecker), the tool was first released in 2019 as an open-source link verification script written in Python, using the requests and BeautifulSoup libraries. Its primary goal is to help webmasters identify 404 errors, redirect chains, and internal linking issues, feeding results into a local report or online dashboard; it is not associated with any commercial search engine, AI training pipeline, or advertising network.

🌐 Technical Behavior

The selflinkchecker bot executes a sequential crawl of specified URLs, following only same‑domain links by default, with a configurable delay of 1 to 5 seconds between requests (typically 2 seconds). Based on the source code on GitHub, it uses a single user‑agent string and does not support concurrent threading by design. The crawler sends standard HTTP/HTTPS GET requests without cookies or JavaScript rendering, checking response status codes and timing out after 10 seconds. IP ranges are not fixed; the bot originates from the IP address of the machine running the check, which could be a home broadband or cloud VM. The tool respects the robots.txt by parsing it via the urllib.robotparser module, but the default configuration does not automatically enforce disallow rules — the operator must enable the “respect robots.txt” flag in the command line arguments. Known crawler cycles are short, typically finishing within minutes for small sites (under 1,000 pages).

📋 robots.txt Compliance

According to the project’s documentation and the source code (github.com/selflinkchecker/selflinkchecker/blob/master/selflinkchecker.py), the tool includes a --respect-robots flag that, when set, causes the crawler to read the domain’s robots.txt and honor all Disallow directives. However, by default this flag is disabled because the author states many website owners want to check all URLs for broken links even if disallowed. For threat intelligence purposes, operators should assume the bot may ignore robots.txt unless explicitly configured otherwise. There are no documented cases of the bot being used for malicious scraping — it is intended as a diagnostic tool for the site owner.

🔍 Detection Indicators

The User-Agent string is selflinkchecker/1.0 (or selflinkchecker/2.0 in later versions) and is not randomized. The bot also sends an X-Requester header set to selflinkchecker as a self‑identification marker (source: GitHub issues). Behavioral fingerprints include a low request rate (average 0.5–1 request per second), no referrer header, and a consistent IP address for a given session. No other known User‑Agent variants exist.

📊 Data Usage

The collected data is used solely by the website owner or the operator of the selflinkchecker instance to generate a list of broken links, status codes, and redirect chains on their own domain. No data is sent to a third‑party server unless the operator explicitly configures an export to a remote endpoint (e.g., via the --output-json flag). The project’s repository states that “no telemetry, analytics, or tracking is built into the tool.”

⚙️ Rate Limiting Policy

Because selflinkchecker is often run by site owners themselves on their own infrastructure, rate‑limiting is not typically required. However, if the bot appears from an unknown IP and ignores robots.txt, administrators may apply a conservative rate limit (e.g., 5 requests per minute per IP) to avoid impact on server resources. The rationale is that a misconfigured instance could accidentally escalate requests, though the tool’s default low speed makes it non‑aggressive.

⚠️

Your Site May Be Hemorrhaging Revenue to Bots

Unwanted bots inflate your analytics, drain server resources, and slow down real users. Check if your site is affected — completely free.

Check My Site for Free

Free to start  ·  Cancel anytime

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.

✓