🤖 Overview

BizWorks Retriever is a web crawler operated by BizWorks Inc., a company specializing in AI-driven business intelligence and data enrichment. First publicly documented in 2022, its primary purpose is to collect publicly available business data—including company descriptions, product listings, and contact information—to feed the BizWorks AI platform, which provides insights for sales, marketing, and competitive analysis. The bot systematically indexes web pages from corporate websites, e-commerce sites, and industry directories, as described in their official documentation at bizworks.com/crawler.

🌐 Technical Behavior

This crawler uses a custom HTTP client that sends requests with a configurable interval, typically ranging from 1 to 10 seconds between pages. It employs a breadth-first crawling strategy, starting from seed URLs provided by the BizWorks system. IP ranges are drawn from a pool of datacenter IPs owned by BizWorks, primarily registered under ASN 397373 (BizWorks Inc.) and distributed across US and European data centers. It supports both HTTP/1.1 and HTTP/2 protocols and requests HTML, JSON-LD, and structured data formats. The bot does not execute JavaScript or render pages, focusing solely on static content. Crawl frequency per domain can be limited algorithmically based on server response times, as per their technical whitepaper.

📋 robots.txt Compliance

Official documentation from BizWorks states that the BizWorks Retriever fully honors robots.txt Disallow directives. It checks the file at the root of each domain before crawling and respects both user-agent-specific and wildcard rules. The bot also adheres to Crawl-delay directives when specified. However, aggressive crawling without delay settings may still occur if no restrictions are defined, a behavior noted in user reports on webmaster forums.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; BizWorksRetriever/1.0; +https://bizworks.com/bot). Additional identifying headers include a custom X-BizWorks-Bot: 1 header and a From header containing a contact email ([email protected]). The bot periodically rotates User-Agent strings but always includes the word "BizWorks" or "BizWorksRetriever". Behavioral fingerprints include consistent request intervals and no referrer headers, making it distinguishable from human traffic.

📊 Data Usage

Collected data is used to train and improve BizWorks' proprietary AI models that power its business intelligence platform. The information is aggregated, cleaned, and structured into a searchable database for clients, enabling use cases such as lead generation, market analysis, and company profiling. Data is not sold directly but used within the BizWorks ecosystem, as confirmed by their privacy policy.

⚙️ Rate Limiting Policy

Due to its bulk data collection approach, the BizWorks Retriever can impose significant load on web servers if left unrestricted. Rate limiting is recommended to protect server resources and maintain service quality, with a threshold of 100 requests per minute per IP being a common safe boundary documented in their best practices guide. The policy rationale is to prevent any single bot from overwhelming infrastructure while still allowing legitimate data collection.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.