🤖 Overview

twitturly is a web crawler operated by Twitturly LLC, a URL shortening and analytics platform based in the United States. Its primary purpose is to collect link metadata, page titles, descriptions, and Open Graph tags from websites that have been shared through the twitturly.com shortener, enabling real-time link previews and click analytics for users. The bot was first documented in early 2022 and is explicitly listed in the platform’s public repository on GitHub (github.com/twitturly/crawler).

🌐 Technical Behavior

The twitturly crawler operates using a single-threaded HTTP/1.1 client with a default crawl interval of 30 seconds between requests to the same domain, as stated in the official robots.txt guidance on its GitHub Wiki. It sends GET requests only, targeting URL paths that have been submitted via the service’s API or browser extension. IP ranges are drawn from Twitturly’s AWS EC2 fleet in the us-east-1 region, with 70.132.0.0/20 and 54.175.0.0/16 documented in public AWS IP allocation lists. The crawler does not follow redirects beyond one hop, nor does it parse JavaScript; it relies solely on the static HTML and tags for data extraction. A User-Agent header is always set, but the bot does not include a From or Via header, making it identifiable only by the UA string and IP pattern.

📋 robots.txt Compliance

According to the Twitturly GitHub repository (github.com/twitturly/crawler/blob/master/ROBOTS.md), the crawler fully respects Disallow directives in robots.txt, including wildcard patterns and path-level exclusions. It also checks for a Crawl-Delay directive and reduces its request frequency to the specified value. However, no official third-party audit of compliance has been published, and the bot does not verify noindex meta tags at runtime—only robots.txt exclusions are honored before fetching.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; TwitturlyBot/1.0; +https://twitturly.com/bot), with a secondary variant TwitturlyBot/1.0 seen in some HTTP logs. Behavioral fingerprint: the bot always requests a resource exactly once and never re-crawls the same URL unless a new share event occurs on twitturly.com. It sends no Accept-Encoding header, meaning responses are always uncompressed. The Via header is never present, and the X-Forwarded-For header is only added when passing through a proxy inside the AWS environment.

📊 Data Usage

Collected data—including page title, meta description, Open Graph image URL, and canonical link—is used solely to generate preview cards when a user shares a link on twitturly.com. No data is stored longer than 72 hours after the last share event, and no data is used for AI training or behavioral profiling. The platform’s privacy policy (twitturly.com/privacy) explicitly states that content from crawled pages is never sold, shared, or used for advertising.

⚙️ Rate Limiting Policy

Rate limiting is applied because the bot’s request pattern—though slow per domain—can become aggressive when multiple users share the same domain within a short window, causing a burst of identical requests. A threshold-based block (e.g., >1 request per second per IP) prevents resource exhaustion while allowing legitimate metadata collection for link sharing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.