🤖 Overview

Google-Extended is a dedicated web crawler operated by Google LLC, introduced on September 28, 2023, as part of Google Search’s infrastructure. Its primary purpose is to collect publicly available web content for training and improving Google’s large language models (LLMs), including those powering Bard (now Gemini) and other AI services. Unlike the main Googlebot used for search indexing, Google-Extended is specifically designed to separate AI training data from search ranking data, giving site owners granular control via robots.txt. According to Google’s official documentation (developers.google.com), it was launched in response to publisher concerns about unlicensed data use for AI.

🌐 Technical Behavior

Google-Extended crawls using the same underlying infrastructure as Googlebot, including HTTP/1.1 and HTTP/2 protocols, with requests originating from Google’s canonical IP ranges published in the googlebot.json file (available at developers.google.com/search/apis/ipranges). Crawl frequency is typically moderate, varying by site authority and freshness requirements—Google states it respects the same rate limits as Googlebot. User-agent verification can be performed via reverse DNS lookups on the googlebot.com domain. The crawler follows standard HTTP headers (Accept, Accept-Language, User-Agent) and sends requests with a randomized delay to avoid overwhelming servers. Google’s official IP ranges for Googlebot (including Google-Extended) are published at https://developers.google.com/static/search/apis/ipranges/googlebot.json, containing over 4,000 IPv4 and IPv6 prefixes. Unlike search indexing crawlers, Google-Extended does not store cached copies of pages for search results.

📋 robots.txt Compliance

Google-Extended fully honors robots.txt directives, as documented in Google’s official announcement (blog.google/products/search/google-extended). Site owners can block it entirely by adding User-agent: Google-Extended Disallow: / to their robots.txt file, and it also respects per-path rules. Google confirmed that existing Disallow rules for Googlebot do not automatically apply to Google-Extended—the two user agents are treated independently. This compliance is verified by Google’s own testing tools and third-party audits (e.g., Cloudflare Radar).

🔍 Detection Indicators

The definitive User-Agent string is: Mozilla/5.0 (Linux; Android 11; Pixel 5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36 followed by Google-Extended. However, Google also provides a simpler token: Google-Extended alone (without browser emulation). Request headers include From: googlebot(at)googlebot.com and User-Agent: Google-Extended. Behavioral fingerprints: reverse DNS resolves to *.googlebot.com, and requests originate from Google’s AS15169 (also known as Google LLC). No specific CVE entries exist for Google-Extended as it is not a vulnerability; it is a legitimate service.

📊 Data Usage

Collected data is used exclusively to train and improve Google’s generative AI models, such as Gemini (formerly Bard), Vertex AI foundation models, and other machine learning systems. According to Google’s privacy policy (policies.google.com), publicly available web text is processed to improve language understanding, summarization, and generation capabilities. Data is not used for advertising personalization or search ranking. Google also offers an opt-out via the Allow directive for whitelisting specific paths if desired.

⚙️ Rate Limiting Policy

Google-Extended is rate-limited to protect server stability and fair resource usage, adhering to the same crawl budget as Googlebot. Administrators should apply threshold-based blocking only when the crawler causes performance degradation, as documented in Google Search Central guidelines (developers.google.com/search/docs/crawling-blocking).

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.