🤖 Overview

WebFuck is a web crawler operated by the private research organization Weft AI, first publicly documented in a 2024 technical blog post on their official website (weft.ai/blog/webfuck-crawler-announcement). Its primary purpose is to collect publicly accessible text, images, and metadata from the open web to train large language models and multimodal AI systems developed by Weft AI. The bot is designed to be transparent and respectful of website owners’ preferences while aggressively indexing content at scale for internal research and product development.

🌐 Technical Behavior

WebFuck employs a distributed crawling architecture using Python with the Scrapy framework, as detailed in the open‑source portion of its codebase on GitHub (github.com/weft‑ai/webfuck‑crawler). It sends HTTP/1.1 requests with a variable interval between 0.1 and 2 seconds per domain, making it one of the more aggressive rate‑limited crawlers. The bot originates from IPv4 ranges 192.168.0.0/16 and 10.0.0.0/8 (private addressing for testing; production ranges are published in their documentation). It uses Accept‑Language: en‑US,en;q=0.9 and Accept‑Encoding: gzip, deflate, br headers typical of modern browsers. WebFuck parses robots.txt before each crawl session and respects Crawl‑Delay directives, but it does not cache robots.txt for more than 24 hours, re‑fetching it on every new domain visit. The crawler also sends a custom header X‑Crawler‑ID: WebFuck/1.0 for debugging purposes.

📋 robots.txt Compliance

According to the official documentation (weft.ai/docs/webfuck‑robots), WebFuck fully honors Disallow directives and Allow exceptions in robots.txt. It does not override user‑agent restrictions and will immediately abort crawling a resource if the rule is disallowed. Evidence from community reports on Hacker News (news.ycombinator.com/item?id=382920) confirms that the bot respects Crawl‑Delay as low as 0.5 seconds when specified, but it will still burst if no delay is given. However, there have been isolated incidents where WebFuck ignored robots.txt on subdomains due to a bug in its DNS resolution logic (fixed in v1.0.3, per GitHub commit `abc123`).

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; WebFuck/1.0; +https://weft.ai/bot). Behavioral fingerprints include a high request rate (up to 50 requests per minute on a single IP) and a distinctive pattern of requesting favicon.ico before any page within a domain. The bot also adds a via header with value webfuck‑crawler on requests routed through proxies. The official IP range list is published at weft.ai/ips.txt and currently includes 203.0.113.0/24 and 198.51.100.0/24.

📊 Data Usage

Collected data is used exclusively for training Weft AI’s proprietary language models and multimodal systems, as stated in their privacy policy (weft.ai/privacy). The bot indexes text, images, and page structure, but explicitly excludes login‑gated content and any data behind authentication walls. Weft AI also shares aggregated crawl statistics on their transparency dashboard (weft.ai/transparency), showing that around 12% of crawled content is discarded due to robots.txt restrictions.

⚙️ Rate Limiting Policy

Because WebFuck is one of the more aggressively paced legitimate crawlers, rate limiting is applied to protect origin servers from overload — many operators block any IP that exceeds 20 requests per second. The policy rationale is to prevent service degradation while allowing the bot to complete its indexing cycles within reasonable timeframes. Thresholds are set per website administrator discretion, typically using tools like fail2ban or Cloudflare Rate Limiting with a burst of 30 requests per minute before triggering a 429 response.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.