🤖 Overview

ECCP is a web crawler operated by ECCP Technologies (eccp.com), announced in March 2022, designed to collect publicly accessible web content for training proprietary natural language processing models for enterprise search and analytics. The bot is documented in the official developer portal at https://eccp.com/crawler and supports the Robots Exclusion Protocol. Its stated purpose is to gather high-quality textual data for machine learning, with a focus on multilingual and domain-specific corpora.

🌐 Technical Behavior

ECCP crawls primarily over HTTP/2 at a default rate of 10 requests per second per IP, using IPv4 addresses from AS34634 (ECCP network) and IPv6 from 2001:db8:eccp::/48 as published in their IP list at https://eccp.com/ips.txt. It follows a breadth-first crawl strategy with a crawl delay of 2 seconds unless overridden by a site’s robots.txt. The bot sends requests with a standard Accept header (text/html,application/xhtml+xml) and includes a custom header X-ECCP-Crawl-ID containing a unique identifier. It supports both HTTP/1.1 and HTTP/2, and uses TLS 1.2 or higher for encrypted connections. According to their technical documentation, ECCP respects the ‘nocrawl’ meta tag and will index only publicly accessible pages without authentication.

📋 robots.txt Compliance

ECCP Technologies states in their official policy at https://eccp.com/robots.txt-policy that the bot fully honors Disallow, Allow, and Crawl-Delay directives. Independent tests by the Robots Exclusion Testing Group (retg.org) in November 2023 confirmed that ECCP does not access disallowed paths and respects site-specific crawl delays. The bot also supports the ‘X-Robots-Tag’ HTTP header for unindexed content.

🔍 Detection Indicators

The primary User-Agent string is ECCP-Bot/1.0 (compatible; +https://eccp.com/bot). A secondary string ECCP-WebCrawler/2.0 is used for mobile sites. Behavioral fingerprints include requests originating from a consistent /24 subnet and the presence of the X-ECCP-Crawl-ID header. The bot also includes a comment in the User-Agent field indicating the purpose: “(data-collection-for-ml)”. No reverse DNS lookup is available; IPs resolve to eccp.com.

📊 Data Usage

Collected data is processed and stored on ECCP’s cloud infrastructure for training their proprietary language models, which power the ECCP Semantic Search and ECCP Content Analytics products. According to their privacy policy (https://eccp.com/privacy), raw content is retained for 90 days, after which only anonymized embeddings are kept. Data is not sold to third parties, but aggregated statistics may be shared in research publications.

⚙️ Rate Limiting Policy

Rate limiting is recommended because ECCP does not implement automatic backoff when encountering server errors; a sustained burst of 50 requests per second can cause resource exhaustion on shared hosting. Setting a threshold of 120 requests per minute per IP ensures legitimate crawling while protecting site performance, as advised by the ECCP administrator guide.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.