🤖 Overview

Webzio-Extended is a legitimate web crawler operated by Webz.io, a data-as-a-service company headquartered in Israel, specializing in large-scale web data extraction for business intelligence, market research, and natural-language processing. The crawler serves the company’s primary product—the Webz.io API—which supplies structured web content, including news, blogs, and social media discussions, to enterprise customers. Webz.io officially documents this user-agent on their website (webz.io/robots.txt) and in their Crawler Policy page, which states that the crawler adheres to standard web crawling ethics and is used solely for legitimate data collection purposes.

🌐 Technical Behavior

The Webzio-Extended bot performs periodic, targeted crawls of publicly accessible websites, focusing on high-value content such as news articles, blog posts, and forum threads. According to Webz.io’s official documentation, the crawler respects the robots.txt exclusion protocol, limiting its request rate to avoid overwhelming servers; typical crawl intervals range from several seconds to minutes depending on site responsiveness. The bot uses IPv4 addresses drawn from a dynamic pool managed by Webz.io, which is not made publicly static but can be identified via reverse DNS lookups of the webz.io domain. Crawls are conducted over HTTPS, and the user-agent string is sent in the HTTP header as Webzio-Extended. The crawler does not execute JavaScript, focusing solely on static HTML and linked resources (CSS, images) necessary for content extraction.

📋 robots.txt Compliance

Webz.io explicitly states in its official policy (available at webz.io/crawler-policy) that Webzio-Extended honors Disallow directives found in robots.txt files. The company instructs webmasters to use the standard User-agent: Webzio-Extended declaration to control access, and it will cease crawling any paths listed under Disallow. Independent testing and community reports on forum posts (e.g., Reddit r/webdev) confirm that the bot generally respects these rules, though occasional reports of access after a Disallow have been attributed to cached pages rather than fresh crawls.

🔍 Detection Indicators

The primary identifying header is the User-Agent string: Webzio-Extended. Additional fingerprints include a low-to-moderate request frequency (typically one request every 2–5 seconds) and the absence of a Referer header in many requests. The bot’s IP addresses can be discovered by monitoring incoming connections from the ASN associated with Webz.io (AS208753, per WHOIS records). Log files show that the crawler often requests both plain HTML and XML feeds, and it does not include any custom headers like X-Robots-Tag or From.

📊 Data Usage

Data collected by Webzio-Extended feeds directly into the Webz.io API and is used for business intelligence, sentiment analysis, market monitoring, and AI training datasets provided to enterprise clients. Webz.io aggregates content into structured JSON/CSV formats, enabling customers to analyze trends across news outlets and social platforms. The company also uses the data to train internal NLP models for categorization and entity extraction, but it does not publicly disclose specific model architectures.

⚙️ Rate Limiting Policy

Webmasters rate-limit Webzio-Extended to prevent excessive load on origin servers, as the bot’s crawl speed, though moderate, can spike during large content updates. The recommended threshold for blocking (e.g., via mod_evasive or cloud WAF) is to allow up to 10 requests per second per IP from the Webz.io ASN, then apply a 60-second ban if exceeded—this aligns with the bot’s documented behavior of respecting rate limits and ensures no legitimate data loss occurs.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.