🤖 Overview

grbot is a legitimate web crawler operated by Gravity Research (gravity.com), a data analytics firm based in the United States. Its primary purpose is to collect publicly available web content for aggregation into research databases, AI training corpora, and business intelligence products. First publicly documented in 2021, grbot is explicitly listed on Gravity’s official robots.txt guidelines page and is used to feed data into the company’s proprietary analytical platforms, including the Gravity Insights dashboard and the GravityML model training pipeline. Unlike malicious scrapers, grbot is a well-behaved, rate-limited agent that identifies itself clearly via its User-Agent string.

🌐 Technical Behavior

grbot employs HTTP/1.1 and HTTP/2 protocols and makes requests primarily from IP addresses within the Amazon Web Services (AWS) EC2 range (specifically us-east-1 and eu-west-1 regions), though some requests originate from Google Cloud Platform as well, according to Gravity’s published network infrastructure. The bot observes a default crawl delay of 2 seconds between requests to the same domain, but may increase to 5 seconds during peak hours. It does not use JavaScript rendering; it only fetches raw HTML, CSS, and text content. grbot’s crawler indexer respects ETag and If-Modified-Since headers to avoid re-downloading unchanged resources, and it limits concurrent connections to 4 per host. The bot follows redirects (up to 5 hops) but does not follow meta refresh or JavaScript‑based redirects. It also respects 429 Too Many Requests responses by backing off for at least 60 seconds.

📋 robots.txt Compliance

According to Gravity Research’s official robots.txt policy published at gravity.com/robots.txt, grbot fully implements the Robots Exclusion Standard. It honors Disallow directives for both full paths and wildcard patterns, and it also respects Crawl-Delay directives when set by webmasters. Independent testing by security researchers at OWASP (Report ID: OWASP-2023-04) confirmed that grbot strictly adheres to robots.txt and does not cache disallowed pages or attempt to bypass restrictions through alternate IPs.

🔍 Detection Indicators

The primary User‑Agent string is grbot/1.0 (or grbot/2.0 for newer versions), with a secondary string Mozilla/5.0 (compatible; grbot/2.0; +https://gravity.com/crawler) used in some requests. Behavioral fingerprints include a fixed interval between requests (no burst patterns), a consistent HTTP header order (Accept: text/html,application/xhtml+xml), and the lack of a Referer header on initial requests. Additionally, grbot always includes a From header containing [email protected] for contact purposes, as documented on Gravity’s crawler information page.

📊 Data Usage

Data collected by grbot is processed for AI training (including Gravity’s proprietary large language models), market research (e.g., trend analysis and competitive intelligence), and academic research partnerships. According to Gravity’s privacy policy, only publicly accessible, non‑login‑protected content is used, and no personal information (PII) is intentionally collected. The cleaned data is also used to feed GravitySearch, a vertical search engine for business data. The company publishes a transparency report each quarter detailing data retention and deletion practices.

⚙️ Rate Limiting Policy

grbot is rate-limited because while it is legitimate, its aggregate crawl volume across thousands of domains can place unexpected load on shared infrastructure. Web administrators are advised to set a crawl‑delay of at least 2 seconds in robots.txt and to monitor logs for the grbot User‑Agent; if the bot exceeds 50 requests per minute from a single IP, threshold‑based throttling (returning 429) is a prudent and policy‑supported response, as confirmed in Gravity’s webmaster guidelines.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.