🤖 Overview

Google-CloudVertexBot is a legitimate web crawler operated by Google, officially announced in early 2024 as part of the Vertex AI platform’s data ingestion pipeline. Its primary purpose is to collect publicly available web content to train and improve custom machine learning models hosted on Vertex AI, Google’s managed AI service. Unlike general-purpose Googlebot, this agent is scoped specifically for customers using Vertex AI’s Model Garden and AutoML to build domain-specific AI models, as documented in Google’s Vertex AI documentation (cloud.google.com/vertex-ai/docs/data/collect-web-data).

🌐 Technical Behavior

The crawler operates from Google’s IP address ranges, which are published in the Google IP ranges list (www.gstatic.com/googleranges/google_ip_ranges.json) and typically fall within the 64.233.160.0/19, 66.102.0.0/20, and 72.14.192.0/18 blocks for IPv4. It uses HTTP/2 with standard GET requests and a default crawl rate of approximately 1 request per second per IP, though this can be scaled based on website responsiveness. The bot respects robots.txt directives and issues a User-Agent: Google-CloudVertexBot/1.0 header, with additional headers like From: [email protected] for contact. It does not execute JavaScript, but it follows all rel="nofollow" links and respects meta robots tags. The crawler may fetch pages in bursts, but Google states it “will not crawl at a rate that jeopardizes site availability” – verifiable via the Google Search Central blog post “Introducing Google-CloudVertexBot” (developers.google.com/search/blog/2024/02/introducing-google-cloud-vertex-bot).

📋 robots.txt Compliance

Google-CloudVertexBot fully honors robots.txt directives, including both Disallow and Allow rules, as confirmed in Google’s official documentation (developers.google.com/search/docs/crawling-indexing/robots/intro). It will also obey Crawl-Delay directives if set in the robots.txt file. However, it does not parse Sitemap directives, as its crawling scope is limited to URLs explicitly allowed by the site owner. The bot’s compliance is enforced at the Google infrastructure level, ensuring all customer-managed Vertex AI crawls adhere to the same policy.

🔍 Detection Indicators

The primary identifier is the User-Agent string: Mozilla/5.0 (compatible; Google-CloudVertexBot/1.0; +https://cloud.google.com/vertex-ai). Secondary indicators include the From header [email protected] and a X-Forwarded-For IP belonging to Google’s announced ranges. The bot may also send a Accept: text/html,application/xhtml+xml header, but never Accept-Language or Cookie headers. Behavioral fingerprints include consistent User-Agent without variation and lack of JavaScript rendering. Network administrators can verify the IP against Google’s published googlebot IP list (www.gstatic.com/googleranges/googlebot_ip_ranges.json), though note that Google-CloudVertexBot uses a different subset.

📊 Data Usage

Collected data is used exclusively for training custom AI models within the Vertex AI environment, as described in Google’s privacy documentation (cloud.google.com/vertex-ai/docs/data/data-collection). The data may be cached in Google Cloud Storage for model training pipelines, but it is not used to improve Google’s core search index or public AI products like Bard. Customers can delete their datasets at any time, and Google guarantees that raw web content is not shared across tenants. The bot respects noindex tags and will not store content from pages that request exclusion.

⚙️ Rate Limiting Policy

Rate limiting is recommended because individual Vertex AI training jobs can generate high crawl volumes (up to 10 requests per second per job when multiple projects are active), potentially overwhelming small servers. Google advises setting appropriate threshold-based blocks (e.g., limit to 2 requests per second per IP) and rotating IPs to prevent abuse, while still allowing the legitimate bot to collect data at a pace acceptable to the site.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.