🤖 Overview

verbstarbot is a web crawler operated by Verblio Inc. (formerly BlogMutt), a content marketing platform headquartered in Austin, Texas. Its primary purpose is to index publicly accessible web content—including blog posts, articles, and product pages—to support Verblio’s AI-driven content analysis, competitive intelligence, and writer quality assurance tools. The bot was first documented in Verblio’s public robot‑BOT policy circa 2019 and is explicitly listed in the company’s official documentation at https://www.verblio.com/bot.

🌐 Technical Behavior

verbstarbot performs both broad and targeted crawls using a headless browser environment simulated by Puppeteer or Playwright, which allows it to execute JavaScript and render dynamic page elements. According to Verblio’s published crawl policy, the bot respects a default crawl delay of 1 second between successive requests, though this may increase under load. It originates from IP addresses belonging to Amazon Web Services (AWS EC2 us-east-1 and us-west-2) as well as DigitalOcean datacenters. The bot communicates exclusively over HTTPS/1.1 and HTTP/2, with TLS 1.2 or higher required. It sends a standard Accept header of text/html,application/xhtml+xml and includes a From header containing the email [email protected] for opt‑out requests. Crawl frequency is typically low to moderate, aiming for no more than 10 requests per minute per domain, as verified by multiple webmaster forum reports.

📋 robots.txt Compliance

verbstarbot fully honors the Robots Exclusion Protocol—it will not crawl any resource matched by a Disallow directive in /robots.txt, as stated in Verblio’s official bot page. The company explicitly instructs operators to use robots.txt to block access, and there are no documented cases of the bot ignoring such rules. A sample directive User‑agent: verbstarbot Disallow: /private/ is provided in their documentation, confirming compliance.

🔍 Detection Indicators

The primary User‑Agent string is verbstarbot/1.0 with the suffix (+https://www.verblio.com/bot). A secondary variant verbstarbot/2.0 has been spotted in server logs since early 2023, though it remains undocumented. Behavioral fingerprints include rapid sequential requests to the same host followed by a pause of exactly 60 seconds, and the presence of a Referer header pointing to https://www.verblio.com. No specific X‑Robot‑Tags are sent, but the bot respects meta name="robots" content="noindex, nofollow" tags.

📊 Data Usage

Verblio uses data collected by verbstarbot for three core functions: (1) training internal natural language processing models that evaluate content quality, readability, and originality; (2) powering a competitive analysis dashboard available to Verblio subscribers; and (3) identifying trending topics and keyword gaps in client industries. The data is aggregated and does not store personally identifiable information, per Verblio’s privacy policy. Retrieved content is cached for up to 30 days before being overwritten.

⚙️ Rate Limiting Policy

Rate‑limiting of verbstarbot is justified because even a low‑frequency crawler can accumulate significant requests over a long crawl cycle, potentially degrading server performance. A threshold of 10 requests per 60 seconds per IP is recommended by Verblio’s own documentation; exceeding this triggers a 429 Too Many Requests response, at which point the bot will back off and reschedule its crawl for a later date.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.