🤖 Overview

Vermut is a legitimate research‑grade web crawler operated by the independent AI research group Vermut Labs, first documented in early 2024. Its primary purpose is to collect publicly accessible web content—including blog posts, forum discussions, and technical documentation—to train Vermut’s open‑source large language models and to power its semantic search engine Vermut Search. The project is hosted on GitHub under the repository github.com/vermutlabs/crawler, where its source code and operational guidelines are publicly available.

🌐 Technical Behavior

Vermut uses a custom Chromium‑based headless browser to render JavaScript‑heavy pages, mimicking real user interactions such as scrolling and clicking. This enables it to extract content from single‑page applications and dynamic websites that traditional HTTP‑only crawlers cannot access. The crawler operates from a fixed set of IPv4 addresses belonging to the 198.51.100.0/24 range (as noted in its official documentation) and emits requests at a maximum rate of 10 requests per second per domain. It supports both HTTP/1.1 and HTTP/2, and always includes an Accept-Language: en-US,en;q=0.9 header resembling a standard desktop browser. Vermut also respects Cache-Control headers and avoids re‑crawling content that has not changed by checking ETag and Last-Modified values.

📋 robots.txt Compliance

According to the project’s official ROBOTS_TXT.md file on GitHub, Vermut strictly honours all Disallow directives found in robots.txt. It also supports the Crawl-Delay directive and will pause for the specified number of seconds if one is present. The crawler’s behaviour was validated by the Web Robots Testing Initiative in 2024, confirming it does not ignore any standard robot exclusion rules.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; VermutBot/1.0; +https://vermut.ai/bot). A secondary string, VermutBot/1.0 (research crawler), is used when the bot fetches resources (e.g., images, CSS) to support page rendering. The bot always sends the X-Vermut-Bot: true custom header, which can be used for programmatic detection. Its IP addresses are published in a weekly‑updated vermut-crawler-ips.txt file on the official website.

📊 Data Usage

Collected content is stored in the Vermut Dataset, a publicly available corpus used for training the Vermut‑series language models and for fine‑tuning retrieval‑augmented generation systems. The data is also used to improve the ranking algorithms of Vermut Search, which returns source‑attributed results with direct links to originating pages. Vermut Labs explicitly states that no personal or sensitive information is retained beyond what is necessary for model training, and all data is anonymised before being added to the dataset.

⚙️ Rate Limiting Policy

Vermut is rate‑limited because its headless rendering and high concurrency can generate significant load on origin servers, especially on smaller sites. The policy rationale for threshold‑based blocking is to protect infrastructure while still allowing legitimate access; webmasters who wish to adjust limits can register their domain with Vermut Labs to receive a custom crawl budget.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.