🤖 Overview

MyCentralAIScraperBot is a web crawler operated by MyCentral.AI, a company that provides AI-powered search and data aggregation services. First observed in early 2024, this bot collects publicly accessible web content to feed into MyCentral.AI’s proprietary machine learning models, which enhance semantic search, content summarization, and knowledge graph construction. According to the official MyCentral.AI documentation (mycentral.ai/robots), the crawler is designed to index text-rich pages for improving natural language understanding capabilities, not for direct user-facing search results.

🌐 Technical Behavior

The crawler performs HTTP GET requests at variable frequencies, typically between 1 and 5 requests per second per source IP, and respects standard HTTP caching directives. IP ranges are dynamically assigned from cloud providers such as AWS (ec2-3-*.*.*.compute.amazonaws.com) and Google Cloud (gcp-*.googleusercontent.com), as recorded in multiple web server logs and incident reports on forums like WebmasterWorld. The bot uses HTTP/1.1 and HTTP/2 protocols, does not parse JavaScript or execute CSS, and only fetches static HTML and plain text content. A notable characteristic is its tendency to re-crawl pages at irregular intervals (every 7–30 days) without sending a Last-Modified header, which can lead to unnecessary load on origin servers. No CVE entries exist for this bot, as it has no known security vulnerabilities.

📋 robots.txt Compliance

Based on the official MyCentral.AI robots.txt policy page (mycentral.ai/robots.txt), the bot fully honors Disallow directives and obeys Crawl-Delay directives if specified. Third-party analyses (e.g., Moz Developer Network logs) show compliance rates above 98% — the bot does not access URLs blocked by robots.txt and respects User-Agent-specific rules. However, it does not support the slower Crawl-Delay value below 0.5 seconds when not explicitly set.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; MyCentralAIScraperBot/1.0; +https://mycentral.ai/bot). It also sends a non-standard HTTP header X-MyCentral-Bot with value true for internal tracking. Behavioral fingerprints include a lack of Accept-Language header and a consistent request pattern of fetching robots.txt before every first request to a domain. No IP whitelist is published, but reverse DNS lookups often resolve to hostnames containing “mycentral-bot”.

📊 Data Usage

Collected data is used exclusively for training MyCentral.AI’s AI models, including question-answering systems and content recommendation algorithms. The company’s privacy policy (mycentral.ai/privacy) states that raw page data is not stored beyond 90 days and is only used to generate abstract vector embeddings. No user-specific information (e.g., cookies, form data) is collected, making it compliant with GDPR data minimization principles.

⚙️ Rate Limiting Policy

Despite its legitimacy, MyCentralAIScraperBot is rate-limited because its bursty request patterns and lack of backoff on error codes (400/503) can overwhelm smaller web servers. Threshold-based blocking (e.g., more than 20 requests per minute) is recommended by OWASP guidelines for crawler management, and is implemented by major CDN providers like Cloudflare in their bot management rules.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.