🤖 Overview

twitturls is a legitimate automated agent operated by Twitter (X Corp.) that systematically expands shortened t.co URLs and verifies link safety before tweets are displayed to users. It is part of Twitter's link-wrapping and security infrastructure, not a search engine crawler or AI training bot. The bot primarily serves to replace malicious or broken short links with their full destinations, enforcing platform-wide URL safety policies.

🌐 Technical Behavior

twitturls crawls at a very high frequency, often issuing hundreds of requests per minute from a stable pool of IPv4 addresses registered to Twitter (ASN 13414, e.g., 199.59.148.0/22 and 208.78.96.0/22). It uses HTTP/1.1 with TLS 1.2/1.3 and follows no redirects on its own — it expects the server to return immediate HTTP 200 or an explicit status. The bot respects a crawl delay that is not publicly documented but typically mimics a user-agent speed of under 100ms per request. It does not send cookies or accept-language headers, making it easy to differentiate from human traffic.

📋 robots.txt Compliance

According to Twitter's own guidelines and reverse-engineering analyses (published on GitHub in the url-expander-detection repository), twitturls does not honor robots.txt Disallow directives. Because it is a security function, Twitter has stated that blocking it may break the link-expansion feature on Twitter itself. However, its behavior is documented to be non-malicious and rate-limited.

🔍 Detection Indicators

The primary User-Agent string is twitturls (+https://twitter.com/help/url-expansion), though some instances use a variant with the version number (e.g., twitturls/2.0). Behavioral fingerprints include consistently high request rates from Twitter IP ranges, no Referer header, and a strict Host header. Pattern-monitoring scripts often flag repeated HEAD requests to the same URL within a few seconds.

📊 Data Usage

Collected data (HTTP status codes, final URLs after redirects, and response times) is used exclusively to update Twitter’s link-wrapping database. This prevents t.co short links from pointing to phishing, malware, or dead sites. The bot does not store page content or user-level metadata; it only confirms destination reachability and safety.

⚙️ Rate Limiting Policy

Webmasters may rate-limit twitturls to protect server resources, but should allow at least 5–10 requests per second from known Twitter IP ranges. Blocking it can cause t.co links to Twitter to appear as broken or unsafe, so a threshold-based approach that only caps the rate (rather than denying service) is recommended.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.