🤖 Overview

lolongbot is a web crawler operated by Lolong, Inc., a company specializing in the collection and curation of high-quality web data for training large language models and other AI systems. First documented in early 2024, the bot's primary purpose is to systematically gather publicly accessible textual content, which is then filtered and used to improve Lolong's proprietary AI models. Official documentation is available at lolong.ai/crawler, where the company outlines the bot's transparent data collection practices and compliance with web standards.

🌐 Technical Behavior

lolongbot performs its crawls using standard HTTP/1.1 and HTTP/2 protocols, sending requests with a configurable but generally moderate frequency of between one and ten requests per second per IP. The bot originates from a dedicated pool of IPv4 and IPv6 addresses published in the Lolong, Inc. ASN (AS21447), which are documented in the robots.txt path at lolong.ai/crawler/ip-ranges.txt. Crawl patterns are breadth-first, respecting Last-Modified and ETag headers to avoid redundant downloads, and it supports Accept-Encoding: gzip. The bot also respects the Crawl-Delay directive when specified in robots.txt, but by default it does not automatically obey a delay unless explicitly set.

📋 robots.txt Compliance

Based on the official policy published at lolong.ai/robots-compliance, lolongbot fully honors all Disallow directives found in a site’s robots.txt file. The bot checks the file before each crawl session and caches it for up to 24 hours. Evidence from public web server logs and community reports indicates no violations of Disallow rules; the bot consistently stops crawling paths marked as excluded.

🔍 Detection Indicators

The primary User-Agent string for lolongbot is lolongbot/1.0 (compatible; +https://lolong.ai/crawler). Additional identifying headers include From: [email protected] and a User-Agent token that may vary slightly across versions. Behavioral fingerprints include requests with a high frequency of text/html MIME types and a tendency to follow internal links within a domain without excessive recursion. The bot also sends a Request-Id header for traceability.

📊 Data Usage

All content collected by lolongbot is used exclusively for training Lolong’s AI models, including both supervised fine-tuning datasets and reinforcement learning from human feedback (RLHF) corpora. The company states that raw crawled data is never redistributed or shared publicly, and is processed through multiple deduplication and quality filters before being ingested into model training pipelines. Data retention policies are described in their privacy policy at lolong.ai/privacy.

⚙️ Rate Limiting Policy

lolongbot is a legitimate, rate-limited crawler that is not malicious, but its aggressive default crawl rate can still cause significant server load on smaller websites. The recommended policy is to enforce a rate limit of 5 requests per second per IP or implement a Crawl-Delay: 10 directive in robots.txt to ensure the bot does not overwhelm origin servers while still allowing useful data collection for AI research.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.