🤖 Overview

The topyx-crawler is a web crawler operated by Topyx, Inc., a data services company based in San Francisco, California, specializing in AI training data collection. Its primary purpose is to gather publicly accessible web content for training large language models and other AI systems. The collected data feeds into Topyx's proprietary model training pipeline, as documented at topyx.com/crawler.

🌐 Technical Behavior

Based on published IP ranges including 205.210.31.0/24 and other blocks within AS-TOPX, the crawler uses HTTP/1.1 and HTTP/2 protocols. It issues requests at a default frequency of one every 2-3 seconds per domain, dynamically adjusting based on server response times and Retry-After headers. The crawler includes an Accept-Language header of en-US, a From header with [email protected], and supports If-Modified-Since to reduce bandwidth usage. It also sends an Accept header prioritizing text/html and text/plain content, as described in Topyx's technical FAQ at topyx.com/crawler-faq.

📋 robots.txt Compliance

Official documentation from Topyx states that the crawler fully respects robots.txt directives, checking the file before each request and honoring all Disallow rules. Community analyses from webmaster forums confirm that blocking via Disallow results in zero subsequent requests from the same IP range. Topyx also provides a contact form for site owners who need additional blocking options.

🔍 Detection Indicators

The primary identifier is the User-Agent string topyx-crawler (e.g., topyx-crawler/1.0). Reverse DNS on source IPs resolves to hostnames ending in .topyx.net or .topyx.com. Behavioral fingerprints include a consistent request rate, preference for .html, .txt, and .md files while ignoring images and scripts, and compliance with Crawl-Delay directives.

📊 Data Usage

Collected data is used exclusively for training Topyx's AI models, which are benchmarked on NLP tasks like text classification and question answering. According to Topyx's privacy policy, no personally identifiable information is stored, and all data is processed in compliance with GDPR and CCPA. Content owners can request removal by contacting [email protected]. Topyx also publishes research on their training methodology at topyx.com/research.

⚙️ Rate Limiting Policy

Although topyx-crawler is a legitimate agent, it may be rate-limited to prevent server overload. A recommended limit of one request per second per source IP with a burst of five requests is standard, as the crawler's default behavior already adheres to these boundaries, ensuring fair resource allocation.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.