🤖 Overview

Gemini-Deep-Research is a legitimate web crawler operated by Google LLC, specifically designed to power the Deep Research feature within Google’s Gemini AI platform. First publicly documented in January 2025 alongside the launch of Gemini 2.0, this bot retrieves publicly available web content to generate comprehensive, multi-step research reports. Unlike general-purpose Googlebot, it is tailored to execute iterative, context-aware searches that mimic human research workflows, synthesizing information from dozens of sources. Official details appear in Google’s Gemini Deep Research product page (deepmind.google/gemini/deep-research) and their AI Crawlers documentation (developers.google.com/search/docs/crawlers-api).

🌐 Technical Behavior

Gemini-Deep-Research initiates crawl sessions with a burst of rapid, sequential HTTP requests to gather data for a single research query, typically sending 10–30 requests within a few seconds per search session. Crawling is performed over HTTPS only, using HTTP/1.1 or HTTP/2 protocol. The bot respects standard crawl delays and does not perform concurrent requests beyond a moderate concurrency of 4–8 parallel connections per host. IP addresses originate from Google’s publicly listed ranges (googlebot.json at developers.google.com/static/search/apis/ipranges.json), specifically from IP blocks assigned to Google Cloud Platform and Googlebot (e.g., 66.249.64.0/19). The user-agent string includes the token Google-Cloud-Vertex-AI in addition to Gemini-Deep-Research, and the bot sends a From header with a Google-owned email domain. Session cookies are not accepted, and the bot performs no file uploads or form submissions.

📋 robots.txt Compliance

Gemini-Deep-Research fully adheres to the robots.txt exclusion protocol as defined by Google’s standard. It respects both Disallow and Allow directives and interprets the Crawl-Delay directive if set. According to Google’s official guidance (support.google.com/webmasters/answer/1061943), this bot falls under the category of “other Google crawlers” that inherit the same robots.txt rules applied to Googlebot unless a separate rule for Google-Cloud-Vertex-AI exists. There is no evidence of deliberate policy violations.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Gemini-Deep-Research/1.0; +https://deepmind.google/gemini/deep-research). A secondary token Google-Cloud-Vertex-AI may appear in the request header as User-Agent: Google-Cloud-Vertex-AI Gemini-Deep-Research. Behavioral fingerprints include a low request frequency per IP (under 1 request per second typical) and the use of a consistent Accept header of text/html,application/xhtml+xml,application/xml;q=0.9. The bot does not send a Referer header.

📊 Data Usage

Collected web content is used exclusively to generate structured research reports for Gemini users who invoke the Deep Research feature. The bot fetches and caches page content temporarily in Google’s infrastructure, then passes it through a large language model for summarization and synthesis. Data is not used to train Gemini models directly; instead, it is ephemerally processed per query and discarded after report delivery, as stated in Google’s privacy policy (policies.google.com/privacy).

⚙️ Rate Limiting Policy

While completely legitimate, Gemini-Deep-Research is rate-limited in production environments due to its bursty request pattern during deep research sessions, which can inadvertently spike server load. The recommended policy is a threshold of 100 requests per minute per IP, with a 429 error response for non‑Google IPs since the bot identifies itself via verified reverse DNS (*.googlebot.com).

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.