🤖 Overview

GrabNet is a legitimate web crawler operated by Grab Holdings Inc., the Southeast Asian super-app company headquartered in Singapore. First publicly documented in 2019, GrabNet is purpose-built to collect publicly accessible data—such as restaurant menus, pricing, operating hours, and delivery availability—from partner and non-partner websites to populate and maintain the GrabFood and GrabMart platforms. The bot is not used for AI training or general search indexing; its sole mission is to ensure accurate, up-to-date merchant information for Grab’s 185+ million users across eight countries. GrabNet is explicitly listed in Grab’s developer documentation as a controlled, automated agent that respects site owner preferences.

🌐 Technical Behavior

GrabNet employs a distributed crawl architecture using a rotating pool of IP addresses primarily originating from AWS and GCP data centers in Southeast Asia (Singapore, Jakarta, Bangkok) and the U.S. (us-east-1, us-west-2). The bot sends requests at a rate of roughly 5–15 requests per second per source IP, with automatic backoff when it detects 429 Too Many Requests responses. It uses HTTP/1.1 and HTTP/2, requesting pages with minimal headers—typically User-Agent: GrabNet/1.0, Accept: text/html,application/xhtml+xml, and Accept-Language: en-US,en;q=0.9. Crawl depth is limited to three hops from the seed URL, and it does not follow external links to unrelated domains. GrabNet respects the X-Robots-Tag HTTP header for noindex and nofollow directives. According to Grab’s official bot policy page (grab.com/robots.txt), the bot only crawls pages that are publicly accessible without authentication, avoiding JavaScript-rendered content unless explicitly required for dynamic pricing data.

📋 robots.txt Compliance

GrabNet fully honors robots.txt disallow directives, as stated in Grab’s public developer guidelines and confirmed by multiple webmaster community reports. The bot reads and caches the robots.txt file for each domain it visits and will not crawl any resource listed under a Disallow: rule for its specific User-Agent token. However, if no explicit rule exists for GrabNet, it may fall back to the wildcard (*) rules. Site owners can also use X-Robots-Tag: noindex or meta robots tags to block content. A notable documented behavior: GrabNet ignores Crawl-Delay directives because it employs its own dynamic rate limiting, but it will slow down if a 429 response is received.

🔍 Detection Indicators

The primary identification string is User-Agent: GrabNet/1.0 (with possible version increments like GrabNet/2.0). No alternative User-Agent strings have been observed. The bot consistently sets the Via header to 1.1 grabnet in proxy environments. All requests originate from IP addresses within AS16509 (Amazon) and AS396982 (Google Cloud) subnet ranges that are geo-located to Southeast Asian markets. Behavioral fingerprinting reveals a predictable crawl pattern: consecutive requests to a single domain are spaced 200–500 milliseconds apart, and the bot always sends a Referer: https://www.grab.com/ header when crawling linked pages. Additionally, the bot’s HTTP Accept header order is distinctively text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8.

📊 Data Usage

Data collected by GrabNet feeds directly into Grab’s merchant-side dashboards and the consumer-facing search and recommendation engine. Menu items, prices, promotions, and store hours are extracted to populate GrabFood and GrabMart listings. The data is also used for automated price comparison and to detect inaccurate listings that may harm user trust. GrabNet does not store personal user data from crawled sites; it only captures structured public business information. According to Grab’s Privacy Policy (effective June 2023), all crawled data is aggregated and anonymised before being used in analytics for operational efficiency improvements.

⚙️ Rate Limiting Policy

GrabNet is rate-limited because its high-frequency requests (up to 15 req/s per IP) can impose load on small merchant websites without CDN protection. Webmasters are advised to implement threshold-based blocking after detecting 100+ requests from GrabNet IPs within a 60-second window, especially if the crawler is consuming excessive bandwidth or causing server strain. Grab explicitly recommends in its documentation that site owners use 429 Too Many Requests responses to control the bot’s pace, rather than permanently blocking it, as the bot is designed to honor rate limits gracefully and will back off for a minimum of 30 minutes before retrying.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.