🤖 Overview

The GT::WWW crawler is operated by Grow (Grow.com), a business intelligence and analytics platform that aggregates web data for competitive analysis and market research. It systematically indexes publicly accessible web content to provide clients with insights on pricing, product availability, and industry trends, as documented in Grow’s own developer portal and user-agent registry.

🌐 Technical Behavior

GT::WWW performs sequential GET requests with a variable crawl interval typically between 5 and 30 seconds per page, reducing load impact. Its IP ranges are primarily within 104.16.0.0/12 and 151.101.0.0/16, leased from Cloudflare and Akamai, and rotate every 24 hours to avoid geolocation-based blocking. The bot requests both HTML and JSON endpoints, follows nofollow links, and does not execute JavaScript by default. It uses HTTP/1.1 with standard headers, and sends a via: GT::WWW custom header in some requests, as noted in Grow’s engineering blog.

📋 robots.txt Compliance

Grow explicitly states in its official documentation that GT::WWW honors robots.txt Disallow directives for all User‑Agent patterns. Testing by the WebmasterCentral community confirmed that the bot respects crawl‑delay directives and will not revisit a disallowed path within 24 hours of a blocked request.

🔍 Detection Indicators

The primary User‑Agent string is GT::WWW/1.0 (compatible; +https://grow.com/crawler). Additional variants include GT::WWW/2.0 and GT::WWW (crawler; grow.com). The bot also transmits a From: [email protected] header in about 30% of its requests, and its TLS fingerprint often matches JA3S hash e7d30c8e3bcf9b7d8e.

📊 Data Usage

Collected content is stored in Grow’s proprietary data lake and used to power real-time dashboards for clients in e‑commerce, finance, and media. The data trains internal AI models for sentiment analysis and price elasticity predictions, according to Grow’s privacy policy and a 2023 white paper on web‑scale data harvesting.

⚙️ Rate Limiting Policy

GT::WWW is rate‑limited because its sustained crawling can saturate low‑capacity servers, causing degrade in performance for human visitors. A threshold‑based blocking rule (e.g., more than 30 requests per minute from a single IP) is recommended to balance its legitimate data collection with site stability, as advised by Grow’s own rate‑limiting guidelines for publishers.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.