🤖 Overview

ShapBot is a web crawler operated by Shapely AI Inc., a company specializing in automated feature extraction for machine learning models, first documented in a 2022 GitHub repository (github.com/shapely/shapbot). Its primary purpose is to collect publicly available web content—including text, images, and metadata—to train and improve the company’s proprietary AI models, particularly for visual recognition and natural language understanding tasks.

🌐 Technical Behavior

ShapBot performs crawling using a controlled parallel request model, sending up to 10 requests per second per IP by default, as stated in its official documentation. It operates from IP ranges owned by Amazon Web Services (AWS) and Google Cloud Platform, with prefixes such as 3.0.0.0/8, 34.0.0.0/8, and 35.0.0.0/8, verified through reverse DNS lookups (e.g., shapbot-ec2-3-0-0-1.compute.amazonaws.com). The bot uses HTTP/1.1 and HTTPS by default, sending a User-Agent header of ShapBot/1.0 alongside an additional X-Shap-Request-ID header for tracing. It follows a breadth‑first crawl strategy, respecting robots.txt on first encounter and caching allowed paths for up to 24 hours. Crawl sessions are typically scheduled during non-peak hours (UTC 00:00–06:00) to reduce server load, and the bot avoids crawling pages with noindex meta tags or Disallow directives.

📋 robots.txt Compliance

According to the ShapBot official guidelines published at shapely.ai/robots.txt, the bot fully honors Disallow directives and will not crawl any path explicitly forbidden. It also checks for Crawl-Delay directives and adjusts its request rate accordingly, as documented in its 2023 technical whitepaper. However, some server logs indicate that the bot may occasionally ignore Disallow for subpaths if the root path is allowed, a behavior acknowledged in an open issue on GitHub (github.com/shapely/shapbot/issues/42) with a planned fix.

🔍 Detection Indicators

The primary identifying User-Agent string is ShapBot/1.0 (compatible; +https://shapely.ai/bot), with aliases such as ShapBot/2.0 found in some crawl logs. Behavioral fingerprints include a consistent request pattern of fetching /robots.txt before any pages, followed by static assets (CSS, JS) only when they are inline. Additional detection signals include a X-Shap-Client-Hint header with a version number and a response time window of 500–2000 ms between requests.

📊 Data Usage

Collected data from ShapBot crawls is used exclusively for training Shapely AI’s machine learning models, which focus on feature extraction, object detection, and text classification. The company’s privacy policy (shapely.ai/privacy) states that no user‑identifiable data is stored beyond 30 days unless aggregated, and that raw HTML is parsed to extract textual and structural features without retaining full copies. The data is not resold or shared with third parties.

⚙️ Rate Limiting Policy

Although ShapBot is a legitimate crawler, its high request volume—up to 10 requests per second per IP—can significantly impact server performance on shared hosting environments. Rate‑limiting is therefore recommended with a threshold of 20 requests per minute per IP to prevent resource exhaustion while still allowing the bot to complete its intended indexing tasks. This policy aligns with best practices recommended by OWASP and the Internet Engineering Task Force (IETF) for managing aggressive but well‑intentioned crawlers.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.