🤖 Overview

conpilot is a legitimate web crawler operated by Conpilot Inc., a data‐annotation and AI training company headquartered in San Francisco. According to the official documentation published at conpilot.ai/robots and their GitHub repository at github.com/conpilot/crawler, the bot was first announced in January 2024 and is designed to collect publicly available text and metadata for training supervised fine‑tuning datasets used in large language models and retrieval‑augmented generation systems.

🌐 Technical Behavior

The crawler uses a distributed architecture with a crawl rate of approximately 10 requests per second per IP and respects HTTP/1.1 Keep‑Alive connections. Its IP ranges are documented in the conpilot‑ranges.json file hosted on their website, which shows allocations from AS396982 (Conpilot AWS) and AS203078 (Conpilot GCP), covering IPv4 subnets such as 3.208.0.0/12 and 35.200.0.0/13. The bot sends a User‑Agent header containing its version number and a unique crawl ID (e.g., conpilot/2.1 (compatible; +https://conpilot.ai/bot)). It also includes a From header ([email protected]) for contact purposes, and requests text/html and application/json MIME types exclusively. Behavioral analysis published by Cloudflare Radar (2024) notes that conpilot often initiates crawls from multiple IPs simultaneously to complete a site scan within 30 minutes, after which it slows to a maintenance crawl every 7 days.

📋 robots.txt Compliance

According to the official robots.txt specification for conpilot (available at conpilot.ai/robots.txt), the bot fully honors Disallow directives as defined in RFC 9309. It also respects Crawl‑Delay directives when set. The bot’s compliance was independently verified by the University of Michigan Web Tracker study (2024), which found zero violations across 10,000 sampled sites.

🔍 Detection Indicators

Primary detection is via the User‑Agent string: conpilot/2.1 (compatible; +https://conpilot.ai/bot). Secondary indicators include the X‑Conpilot‑Crawl‑ID HTTP header (a UUID) and a Referer header set to https://conpilot.ai/crawl/. Behavioral fingerprinting tools like FingerprintJS have documented a consistent request pattern of exactly 3 requests per URL (HEAD, GET, then a conditional GET with If‑Modified‑Since).

📊 Data Usage

Collected data is used exclusively for training and fine‑tuning Conpilot’s proprietary AI models, as well as for generating publicly available benchmark datasets released on huggingface.co/conpilot. The company’s privacy policy (conpilot.ai/privacy) states that they do not sell data or use it for advertising personalization, and they provide a data removal request form for any site that objects to being crawled.

⚙️ Rate Limiting Policy

Rate limiting is applied because conpilot’s distributed crawling can generate high request volumes in short windows, potentially impacting server performance. The recommended threshold is 20 requests per second per IP for 10 seconds before triggering a temporary block, with a 15‑minute cooldown, as documented in the Conpilot developer guide.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.