🤖 Overview

contype is a web crawler operated by Contype Inc., a data analytics firm that launched the bot in 2021 to collect publicly available text for training commercial machine learning models. According to the official Contype documentation at contype.io/crawler, its primary purpose is to build high-quality datasets for natural language processing and content classification products.

🌐 Technical Behavior

Contype crawls using a distributed fleet of AWS EC2 instances, with IP ranges documented in the provider’s IP list (54.x.x.x, 52.x.x.x). It makes HTTP/1.1 and HTTP/2 requests at a sustained rate of ~10 per second per IP, with bursts to 20 req/s. The bot follows standard redirects, respects Cache-Control headers, and fetches only HTML and plain text, ignoring images and PDFs. It performs breadth-first traversal from seed URLs, following same-domain links to a depth of 5, and checks sitemap.xml for priority paths. A default 30‑second timeout is enforced, and the bot does not accept cookies or store session data.

📋 robots.txt Compliance

The official Contype documentation at contype.com/robots-txt states that the crawler fully obeys Disallow directives and supports Crawl-Delay. It rechecks robots.txt every 24 hours; if the file is missing, it proceeds with default crawl behavior. The bot also respects Allow overrides and does not crawl URLs listed in disallowed paths.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; ContypeBot/2.0; +http://contype.com/bot). It also sends headers X-Crawler-Id: Contype and From: [email protected]. Requests originate from AWS IP ranges and show a predictable inter‑request interval; server logs often show the string “ContypeBot” alongside the identifying URL. The bot does not spoof other User‑Agents.

📊 Data Usage

The collected data is used internally by Contype Inc. to train proprietary models for text classification, sentiment analysis, and content recommendation. According to their privacy policy at contype.com/privacy, all personal information is anonymized before processing and datasets are not redistributed. The company also uses aggregated statistics to improve its client‑facing analytics dashboard.

⚙️ Rate Limiting Policy

Because ContypeBot can sustain 10 requests per second per IP, rate‑limiting is recommended to protect server resources. A threshold of 5 req/s per IP with a 503 response after exceeding it balances the bot’s data‑collection needs with site performance, as advised in the official Contype rate‑limit guidelines at contype.com/rate-limit.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.