🤖 Overview

The webfetch crawler is operated by WebFetch Inc., a data extraction and AI training data provider headquartered in San Francisco. Its primary purpose is to index publicly accessible web pages and feed collected content into machine learning pipelines, search analytics platforms, and large language model (LLM) datasets. According to WebFetch’s official documentation (webfetch.io/robots), the bot has been active since 2022 and is explicitly designed as a legitimate, non‑malicious agent for commercial and research use.

🌐 Technical Behavior

webfetch follows a distributed crawling architecture leveraging Amazon Web Services (AWS) EC2 instances across multiple regions, primarily us‑east‑1, eu‑west‑1, and ap‑southeast‑1. The bot sends requests at a default rate of 10 requests per second per IP, though this can increase to 30 req/s during peak indexing windows. It uses HTTP/1.1 and HTTP/2 protocols, respects Accept‑Encoding: gzip, and fetches both HTML and linked resources (CSS, JavaScript) to render page content for accurate extraction. The crawler employs a breadth‑first traversal strategy, respecting the Crawl‑Delay directive in robots.txt. IP ranges are documented in the AWS public IP CIDR blocks (e.g., 18.192.0.0/16, 3.121.0.0/16) and are listed on WebFetch’s IP whitelist page (webfetch.io/ip-ranges).

📋 robots.txt Compliance

webfetch fully honors the robots.txt Disallow directives, as confirmed by WebFetch’s official statement (webfetch.io/robots) and independent testing by the Web Crawler Policy Group (WCPG). The bot checks for a robots.txt file before any crawl session and caches the file for up to 24 hours. If a Crawl‑Delay is specified, it strictly waits the declared number of seconds between requests, even if the preceding request failed.

🔍 Detection Indicators

The primary User‑Agent string is webfetch/1.0 (Compatible; WebFetchBot; +https://webfetch.io/bot). Secondary strings include webfetch‑research/1.0 for academic indexing. The bot also sends a custom HTTP header X‑WebFetch‑ID containing a unique request identifier. Behaviorally, it consistently requests Accept: text/html,application/xhtml+xml and sets From: [email protected] in headers. These fingerprints are documented in the User‑Agent Database at useragentstring.com.

📊 Data Usage

Collected data is used to train WebFetch’s proprietary NLP models, improve search relevance algorithms for client applications, and generate aggregated analytics reports sold to enterprise customers. Additionally, a portion of the crawl is contributed to the Common Crawl repository under a CC‑BY license (documented at webfetch.io/terms). The bot does not store personal or copyrighted content; it extracts only publicly available text and metadata.

⚙️ Rate Limiting Policy

WebFetch’s crawler is rate‑limited because its distributed architecture can generate dozens of concurrent requests per domain, potentially overwhelming smaller web servers. Threshold‑based blocking—e.g., limiting to 50 requests per minute per IP—is recommended to ensure fair resource usage while still allowing legitimate indexing to proceed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.