🤖 Overview

AgentTimes is a web crawler operated by AgentTimes Inc., a company specializing in AI training data acquisition, first documented in public robots.txt discussions in early 2023. Its primary purpose is to systematically collect publicly accessible web content for training large language models (LLMs) and other machine learning systems, feeding data into proprietary AI products used for natural language understanding and generation. The bot is explicitly mentioned in the official documentation of major websites like GitHub and Stack Overflow, where it is listed as a allowed or disallowed agent.

🌐 Technical Behavior

The AgentTimes crawler uses HTTP/1.1 and HTTP/2 protocols, sending GET requests with a default crawl rate of approximately one request per 5 seconds per domain, as observed from public web server logs. Its IP ranges are dynamic and drawn from cloud providers such as AWS and DigitalOcean, with reverse DNS records typically resolving to hostnames like crawl.agenttimes.com. The bot employs a depth-first traversal of internal links, respecting rel="nofollow" and noindex meta tags, and avoids login pages and forms. It caches content with ETag headers and re-crawls pages based on a configurable revisit interval, defaulting to 7 days. The crawler’s user-agent string includes version numbers (e.g., AgentTimes/1.0) and is often accompanied by a From header containing an email address ([email protected]).

📋 robots.txt Compliance

According to the official documentation published on agenttimes.com/robots, the crawler fully respects Disallow directives in robots.txt files and will cease crawling URI paths after a 403 or 410 HTTP status. The company provides a public robots.txt validator tool on their site to help webmasters test rules. Compliance is monitored through automated logs, and violations are escalated to the engineering team within 24 hours, ensuring minimal impact on origin servers.

🔍 Detection Indicators

Identifying AgentTimes in web server logs is straightforward through its user-agent string: Mozilla/5.0 (compatible; AgentTimes/1.0; +https://agenttimes.com/crawler). Additional behavioral fingerprints include a low referrer rate (the bot rarely sends a Referer header) and a preference for HTML pages with large text content over binary files (images, PDFs). The bot may also include a X-Robots-Tag header in its requests indicating its identity.

📊 Data Usage

Collected data is used exclusively for training large language models (LLMs) owned by AgentTimes Inc., including a GPT-style model called AT-Model (not publicly released). The company claims that all data is filtered to remove personally identifiable information (PII) and that no user-generated content from paywalled areas is stored. Data retention policies are outlined in their privacy policy at agenttimes.com/privacy, stating a 90-day raw log retention period before aggregation.

⚙️ Rate Limiting Policy

AgentTimes is rate-limited because its continuous crawling—though legitimate—can still trigger resource exhaustion on shared hosting environments. A threshold-based blocking policy (e.g., 50 requests per minute per IP) is justified to protect application performance while allowing the bot to complete its indexing within reasonable timeframes.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.