🤖 Overview

RedCarpet is a web crawler operated by RedCarpet Inc., a company specializing in AI-driven product recommendation engines for e-commerce platforms. Its primary purpose is to collect publicly available product data, pricing, descriptions, and images from retailer websites to train and improve RedCarpet's machine learning models. The bot feeds data into the RedCarpet Smart Shopper platform, which provides personalized shopping experiences.

🌐 Technical Behavior

The RedCarpet bot performs HTTP GET requests with a controlled crawl rate of approximately one request per second per IP address, though it may burst up to five requests during initial indexing. It uses a rotating set of IP addresses sourced from Amazon Web Services (AWS) and Google Cloud Platform (GCP), typically within ranges documented in public ASN databases. The crawler supports HTTP/1.1 and HTTPS, and it sends a User-Agent header that includes versioning. It will follow redirects and respects Cache-Control headers. The bot is designed to avoid deep directories and focuses on product pages, category listings, and sitemaps, with a crawl depth limited to 5 levels from the root.

📋 robots.txt Compliance

According to official documentation on the RedCarpet bot information page at https://www.redcarpet.com/bot, the crawler fully honors robots.txt directives, including Disallow rules. It reads the robots.txt file at the start of each crawl session and caches it for up to 24 hours. Webmasters can block specific paths using standard syntax.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; RedCarpetBot/1.0; +https://www.redcarpet.com/bot). Occasionally, older versions may use RedCarpet/1.0 without the Mozilla prefix. Behavioral fingerprints include a consistent request interval of 1–2 seconds and the absence of JavaScript rendering. The bot does not send a Referer header and does not accept cookies.

📊 Data Usage

Collected data is used to train deep learning models for product recommendation, price comparison, and inventory analysis. The data is stored in RedCarpet's cloud infrastructure and is not shared with third parties. The company also uses the data to generate market insights for retailers participating in its platform.

⚙️ Rate Limiting Policy

Despite being a legitimate crawler, RedCarpet is rate-limited because its distributed architecture can lead to high aggregate request volumes, impacting server performance. Many webmasters apply threshold-based blocking after observing more than 10 requests per second from the bot's IP ranges.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.