🤖 Overview

webgo is a legitimate web crawler operated by WebGO Technologies Ltd., a company specializing in AI-driven data enrichment and market intelligence. First publicly documented in 2021, the bot systematically indexes public web pages to feed the WebGO Data Platform, a service that provides aggregated content for competitive analysis and trend forecasting. According to the official WebGO documentation (webgo.io/bot), the crawler is explicitly designed to respect site owner preferences and is not associated with any malicious activity.

🌐 Technical Behavior

The webgo bot employs a polite crawling strategy with a default request interval of 10 seconds between pages, as specified in its technical overview (webgo.io/crawl-policy). It uses HTTP/1.1 with Accept-Language: en-US,en;q=0.9 and respects If-Modified-Since headers to reduce server load. IP ranges are documented in the official WebGO IP Whitelist (webgo.io/ips), which includes prefixes from AS13335 (Cloudflare) and AS16509 (Amazon), though the bot primarily sources residential proxies to avoid triggering geographic blocks. Crawl depth is limited to 3 levels by default unless the site explicitly allows deeper indexing via a X-Robots-Tag: webgo:max-depth:5 header. The bot also parses JavaScript-rendered content using a headless Chromium instance, as noted in their GitHub repository (github.com/webgo/crawler).

📋 robots.txt Compliance

webgo fully honors Disallow directives in robots.txt, as verified by WebGO’s published compliance statement (webgo.io/robots). The bot checks the file before every crawl and adheres to Crawl-Delay directives, setting a minimum wait of 5 seconds if specified. In independent tests (e.g., bot-tester.com, 2023), no violations were observed.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; webgo/1.0; +https://webgo.io/bot), though variations may include webgo-crawler/2.0 or WebGOBot/3.1. Behavioral fingerprints include a consistent request order (first /robots.txt, then /sitemap.xml, then pages), and the absence of referrer headers. IP ranges are publicly listed at webgo.io/ips.txt.

📊 Data Usage

Collected data feeds the WebGO Intelligence Platform, which offers real-time market trend analysis, competitor pricing snapshots, and sentiment extraction. Data is aggregated and anonymized; no personal information is retained. The platform is used by Fortune 500 companies for strategic planning, as outlined in case studies on webgo.io.

⚙️ Rate Limiting Policy

We rate-limit webgo to 20 requests per minute per IP to prevent excessive load on origin servers while allowing legitimate data collection. The policy is documented in our standard security guidelines (see rate-limits.md), balancing operational availability with the crawler’s published politeness parameters.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.