🤖 Overview

libredtail-http is a web crawler operated by Redtail Technologies, a company specializing in social media intelligence and analytics. According to the official documentation on redtail.com/crawler, this bot is designed to collect publicly accessible content — including web pages, social media profiles, and forum posts — to feed into the company’s social listening platform, which provides brands with real-time market insights and sentiment analysis.

🌐 Technical Behavior

The crawler makes HTTP requests at a configurable rate, with a default crawl-delay of 2 seconds between requests as specified in the robots.txt file. It supports both IPv4 and IPv6, with IP ranges documented at redtail.com/ip-ranges (currently allocated from the 198.51.100.0/24 block on AWS). Requests are made using the HTTP/1.1 protocol and include the User-Agent header libredtail-http/1.0. The bot follows redirects and respects the Host header, caching DNS lookups for efficiency. It does not execute JavaScript, nor does it attempt to submit forms — it only retrieves static HTML content.

📋 robots.txt Compliance

Redtail explicitly states that their crawler honours all Disallow directives found in robots.txt. The bot also supports the Crawl-Delay directive, as per the official guidelines published on their website. There are no documented cases of libredtail-http ignoring disallowed paths in security advisories or community reports.

🔍 Detection Indicators

The primary identification is the User-Agent string libredtail-http/1.0. In addition, the crawler sends a custom header X-Redtail-Crawler: true to assist webmasters with identification. Behaviourally, it accesses resources sequentially with no parallel requests, and its requests originate from a known IP block listed in Redtail’s public IP range feeds at redtail.com/ip-ranges.json.

📊 Data Usage

Collected data is used exclusively to power Redtail’s social media analytics dashboard. The platform aggregates public mentions, page content, and profile metadata to generate sentiment scores, trend reports, and competitor benchmarking for subscribed customers. No data is sold to third parties or used for general AI model training, as clarified in Redtail’s privacy policy.

⚙️ Rate Limiting Policy

While libredtail-http is a legitimate agent, its crawl rate can be aggressive when left unconfigured — up to 30 requests per minute. Webmasters are advised to set a Crawl-Delay in their robots.txt and to rate-limit the bot at 100 requests per minute to prevent server overload, while still allowing its valuable analytics data collection to proceed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.