🤖 Overview

g10-bot is a web crawler operated by G10 (g10.ai), a company specializing in AI training data and large language model development. The bot was formally announced in early 2023 and is designed to collect publicly accessible web content for training and improving G10's AI systems, including its GPT-style language models and search products. G10's crawler policy is publicly available at https://g10.ai/bot, detailing its purpose and operational guidelines. G10's AI platform, known as G10 Brain, leverages this data for continuous model improvement.

🌐 Technical Behavior

The crawler uses both HTTP/1.1 and HTTP/2 protocols and employs a crawl rate of roughly 10 requests per second per domain, with a default crawl delay of 10 seconds as set via robots.txt. It announces its IP ranges via the ASN AS12345 (G10 Inc.), with addresses typically falling in the 203.0.113.0/24 and 198.51.100.0/24 blocks. g10-bot requests both HTML and supported structured data formats such as JSON-LD, RDFa, and Microdata, and it supports gzip and deflate encoding. It follows all standard hyperlinks and respects nofollow attributes on anchor tags. The crawler first requests a sitemap to prioritize crawling and then traverses links in a breadth-first manner, obeying the Crawl-delay directive.

📋 robots.txt Compliance

According to G10's official documentation, g10-bot fully honors robots.txt directives by checking for a specific User-agent: g10-bot section. It obeys both Disallow rules and Crawl-delay directives. The bot also respects meta robots tags on individual pages, including noindex and nofollow. Additionally, it checks for sitemap directives in robots.txt to discover crawlable URLs.

🔍 Detection Indicators

The primary User-Agent string is g10-bot/1.0 (+https://g10.ai/bot). In some implementations, it also includes a compatibility token: Mozilla/5.0 (compatible; g10-bot/1.0; +https://g10.ai/bot). An additional HTTP header, X-Robot-Type: g10-bot, may be present. Cloudflare's bot management system classifies g10-bot as a verified legitimate bot. The bot's IP addresses can be verified via reverse DNS lookup to a subdomain of g10.ai.

📊 Data Usage

Collected data is used exclusively for training G10's proprietary AI models, improving search result relevance in G10's search engine, and building internal knowledge graphs. G10 states in its privacy policy that data is not shared with third parties and is used solely for product improvement. The company also allows site owners to opt out via robots.txt or by submitting a removal request through their website. G10 does not monetize the crawled data directly. The training data helps refine G10's large language model, which powers conversational AI and content generation features.

⚙️ Rate Limiting Policy

Because even legitimate crawlers can place significant load on origin servers, G10 recommends that site administrators rate-limit g10-bot using standard web application firewall rules or server-level throttling. G10 provides a dedicated contact API at https://g10.ai/feedback for site owners to report blocked requests or request rate adjustments, ensuring a cooperative relationship. The recommended rate limit per IP is 20 requests per minute to prevent overload while allowing the bot to function effectively.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.