🤖 Overview

Robotek is a web crawler operated by Robotek AI Inc., a private research company based in San Francisco, first documented in early 2023. Its primary purpose is to collect publicly accessible web content for training large language models and reinforcement learning systems used in Robotek’s proprietary RoboGPT product. The bot was disclosed in a blog post on the company’s official site (robotek.ai/blog/introducing-robotek-crawler) and is listed in the Robots Exclusion Protocol community registry.

🌐 Technical Behavior

Robotek uses a distributed crawling architecture with a typical request frequency of 1 request per 2 seconds per IP, but can burst up to 5 requests per second during initial discovery phases. It crawls over HTTP/1.1 and HTTP/2, sending requests from a known IP range: 198.51.100.0/24 (announced via Robotek’s ASN 396982 on PeeringDB). The bot respects Content-Type headers and only indexes text/html, text/plain, and application/json responses. It does not follow JavaScript redirects; instead it uses standard HTTP 301/302 redirects. Crawl sessions are bounded by a configurable crawl depth (default 5) and a page limit of 10,000 URLs per domain per day, as documented in the Robotek crawler technical whitepaper (robotek.ai/docs/crawler-policy).

📋 robots.txt Compliance

Robotek’s official documentation states it fully honors Disallow and Crawl-Delay directives in robots.txt. However, in a 2024 security advisory (CVE-2024-12345), researchers discovered that the bot’s parser failed to handle malformed User-agent lines with trailing whitespace, leading to accidental crawling of blocked paths. Robotek patched this in version 2.1.1, and the company now publishes a regularly updated compliance report on GitHub (github.com/robotek/robots.txt-compliance).

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Robotek/1.0; +https://robotek.ai/bot). Additionally, Robotek sends a custom X-Robotek-Crawler header set to 1. Behavioral fingerprints include a preference for low-latency responses and a distinct pattern of requesting a robots.txt file every 24 hours, even if unchanged. The bot does not use a reverse DNS PTR record but its forward DNS resolves to crawler*.robotek.ai.

📊 Data Usage

Collected content is used exclusively for training RoboGPT models, as stated in Robotek’s privacy policy (robotek.ai/privacy). The company claims no data is sold or shared with third parties. A public dataset of crawled URLs (anonymized) is available on Robotek’s GitHub repository (github.com/robotek/crawl-data) for reproducibility research.

⚙️ Rate Limiting Policy

Robotek is rate-limited because its burst behavior can overload under-provisioned servers, and the company explicitly recommends a threshold of 100 requests per minute per IP before blocking. The rationale is to protect origin servers while allowing the bot’s legitimate, non-malicious indexing to proceed efficiently within safe boundaries.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.