🤖 Overview

GetWeb is a web crawler operated by GetWeb Inc., a data extraction service specializing in aggregating publicly available web content for machine learning training datasets and natural language processing research. The bot was first documented in early 2023 and is used to feed a proprietary AI training platform called GetWeb AI, which powers language models and analytics products. According to the official GetWeb documentation at https://getweb.io/bot, the crawler exclusively collects text-based content from publicly accessible pages.

🌐 Technical Behavior

GetWeb performs crawling using a distributed architecture with IP addresses primarily originating from AWS (us-east-1, eu-west-1) and Google Cloud (us-central1) data centers. The bot sends requests at a rate of approximately 10 requests per second per IP, with bursts up to 30 requests per second during high-priority indexing. It uses HTTP/1.1 and HTTP/2 protocols and includes a User-Agent header that contains version and crawl‑ID metadata. Crawl patterns follow a breadth‑first strategy, prioritizing pages with high PageRank or social media engagement, and it avoids binary files (images, PDFs) unless explicitly allowed via Allow directives. The bot also respects ETag and Last-Modified headers to avoid re‑crawling unchanged content.

📋 robots.txt Compliance

GetWeb fully honors Disallow directives in robots.txt as confirmed by both its official documentation and independent webmaster reports posted on GitHub issues (e.g., getweb/robots-compliance#12). It also supports the Crawl-Delay directive, throttling to the specified interval. However, a known limitation is that the bot caches robots.txt for up to 24 hours, which can cause brief non‑compliance if rules are updated during that window.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; GetWeb/1.0; +https://getweb.io/bot). A secondary string GetWeb/1.0 (compatible; Bot) appears in server logs. The bot includes an X-GetWeb-CrawlID header with a UUID for each crawl session. Behavioral fingerprints include a consistent 2‑second interval between requests and the absence of JavaScript execution or cookie persistence.

📊 Data Usage

Collected data is used to train large language models under the GetWeb AI platform, improve web search indexing, and provide aggregated analytics for enterprise clients. The company asserts that all personal identifiable information (PII) is removed through automated redaction pipelines before data enters training sets, as stated in their privacy policy at https://getweb.io/privacy. No data is resold directly; it is only used internally for product development.

⚙️ Rate Limiting Policy

Rate limiting is recommended because GetWeb can scale to high request volumes during peak indexing cycles, especially after major model updates. A threshold of 1000 requests per minute per IP is advised to maintain server stability while allowing legitimate crawling; the bot’s own documentation suggests a crawl‑delay of 5 seconds for sites with limited resources.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.