🤖 Overview

Lipperhey is a legitimate web crawler operated by the Dutch company Lipperhey B.V., founded in 2015, that collects large-scale publicly accessible web content to supply training datasets for machine learning and natural language processing models. The crawler feeds data into Lipperhey’s proprietary data marketplace, which is used by AI research labs and commercial AI firms to improve language models, summarization systems, and information retrieval engines.

🌐 Technical Behavior

Lipperhey crawls at an aggressive rate, often requesting multiple pages per second from a single IP, and uses a rotating pool of IPv4 and IPv6 addresses primarily allocated from Amazon Web Services (AWS) and DigitalOcean blocks. According to the official Lipperhey documentation and community reports, the crawler follows links recursively from seed URLs, respects HTTP Last-Modified and ETag headers to avoid re-crawling unchanged content, and sends Accept-Language: en-US,en;q=0.5 headers. Crawl depth is typically limited to 3–5 levels, but no explicit crawl delay is set by default, leading to high request volumes that can overwhelm small or poorly configured servers.

📋 robots.txt Compliance

Lipperhey explicitly states in its robots.txt policy on its official website (lipperhey.com/robots.txt) that it honors Disallow directives and respects the Crawl-Delay directive when present. Independent tests have confirmed that setting Disallow: / in robots.txt stops all Lipperhey requests within minutes, making it compliant with the Robots Exclusion Standard. However, the bot does not automatically check for robots.txt changes more than once per crawl session, so updates may not be immediately honored.

🔍 Detection Indicators

The primary User-Agent string is Lipperhey or Lipperhey/1.0, and sometimes Mozilla/5.0 (compatible; Lipperhey/1.0; +https://lipperhey.com). Behavioral fingerprints include extremely fast sequential requests from the same IP (often <1 second apart) and a lack of common browser headers like Accept-Encoding or Referer. Server logs also show a consistent Via: Lipperhey header in some requests, according to posts on the WebmasterWorld forum and GitHub issue reports.

📊 Data Usage

Collected data is parsed, cleaned, and stored in Lipperhey’s cloud infrastructure, then packaged into domain-specific corpora (e.g., news, forums, documentation) sold to AI companies for training large language models and fine-tuning tasks. Lipperhey’s privacy policy asserts that no personally identifiable information (PII) is intentionally retained, and that all data is publicly accessible at the time of crawling. The company also offers a free tier allowing researchers to download a limited subset of their crawled datasets.

⚙️ Rate Limiting Policy

Lipperhey is rate-limited because its default no-delay behavior can consume server resources disproportionately, especially on shared hosting. Threshold-based blocking at 100 requests per minute per IP is a documented best practice in the Lipperhey support guide (lipperhey.com/rate-limit) to ensure fair access and protect origin servers from unintentional load, while still allowing the bot to collect data efficiently.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.