🤖 Overview

YOURLS is the official HTTP client component of the YOURLS (Your Own URL Shortener) open‑source project, first released in 2011 and maintained on GitHub (github.com/YOURLS/YOURLS). It is not a dedicated crawler but rather a lightweight agent that fetches the content of a single URL when a new short link is created, enabling the service to automatically extract metadata such as the page title, description, and Open Graph image. The bot is operated by each individual YOURLS instance administrator and is not associated with any central organization or commercial product.

🌐 Technical Behavior

The YOURLS agent executes a single HTTP GET request per shortened URL using PHP’s cURL library or the WordPress HTTP API (depending on the environment). It does not follow links, does not parse the document for additional URLs, and does not engage in recursive crawling. The request timeout is configurable, defaulting to 10 seconds. Since the bot runs on the same server as the YOURLS installation, its source IP is the server’s own IP address and therefore not part of any fixed, published IP range. It sends a standard Accept header and, by default, a User‑Agent string of YOURLS/1.x (e.g., YOURLS/1.9.2). The official YOURLS documentation (docs.yourls.org) notes that administrators can override the User‑Agent via the YOURLS_USER_AGENT constant in the configuration file, making detection more challenging.

📋 robots.txt Compliance

YOURLS does not include any built‑in logic to check robots.txt before making a request. The official repository (github.com/YOURLS/YOURLS/blob/develop/includes/functions-http.php) shows that the HTTP fetch function simply issues the request without parsing any exclusion rules. However, because the agent only retrieves a single page per user action, it poses a negligible load on most sites and is not typically considered a problem. Site administrators who wish to block YOURLS requests can rely on the User‑Agent string for filtering.

🔍 Detection Indicators

The primary detection method is the User‑Agent string, which by default follows the pattern YOURLS/ followed by the version number — for example YOURLS/1.9.2. Older versions may use YOURLS without a version suffix. The request also sends a standard Accept header (text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8) and a Accept‑Language header of en‑US,en;q=0.5 unless the admin has customized these. No other unique HTTP headers are sent by default. The Referer header may be absent or set to the YOURLS administration page.

📊 Data Usage

The fetched metadata — primarily the page title, description, and Open Graph image — is stored in the YOURLS database and displayed on the shortened URL’s preview page and in the admin dashboard. This data is not sold, shared, or used for AI training, search indexing, or analytics aggregation. It remains exclusively within the self‑hosted instance. The project’s privacy policy (github.com/YOURLS/YOURLS) states that no external data collection is performed by the YOURLS software itself.

⚙️ Rate Limiting Policy

Because the YOURLS agent only makes a single request per shortened URL, it rarely triggers rate limits. However, administrators who configure YOURLS to automatically fetch previews for imported bulk URLs may inadvertently generate a rapid sequence of requests. In such cases, it is prudent for target sites to apply a conservative rate limit (e.g., 5 requests per second) to protect server resources, as the agent will simply time out and not retry aggressively—the YOURLS HTTP class does not implement exponential backoff.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.