🤖 Overview

The Ai2Bot-Dolma crawler is operated by the Allen Institute for AI (AI2), a non-profit research institute based in Seattle. It is purpose-built to collect publicly accessible web content for the creation of the Dolma dataset, a massive open-source corpus used to train large language models. The project was announced in September 2023 and is part of AI2’s OLMo (Open Language Model) initiative. The bot explicitly excludes paywalled, login-required, or explicitly blocked content.

🌐 Technical Behavior

The crawler respects crawl-delay directives in robots.txt and implements a default delay of 1 second between requests. Its crawl pattern is depth-first, targeting HTML pages primarily, and it does not request JavaScript, CSS, or image files unless explicitly linked as primary content. IP ranges are not publicly fixed but the bot resolves from ec2-* compute.amazonaws.com domains, indicating operation on AWS EC2 instances. The vast majority of requests use HTTP/1.1 with standard headers. The bot also includes a robots.txt check before every single request, re-reading the file on each crawl session.

📋 robots.txt Compliance

AI2’s official documentation states that Ai2Bot-Dolma fully honors all Disallow directives in robots.txt. The bot explicitly avoids crawling any path marked as disallowed and also respects Crawl-Delay and Allow directives. The project’s GitHub repository (github.com/allenai/dolma) contains verification scripts for compliance testing. There are no documented cases of the bot ignoring robots.txt.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Ai2Bot-Dolma/1.0; +https://allenai.org/robots.html). Additional fingerprints include a User-Agent header containing the specific version number, and the bot always includes a From header with the email [email protected]. The absence of a Referer header and the presence of a consistent Accept: text/html header are further indicators.

📊 Data Usage

All collected text content is used exclusively to build the Dolma dataset, which is released under an open license (CC-BY-4.0 or similar) for training AI models. The dataset is used by AI2 for its OLMo series of open-source language models, and is also freely available to the global research community. The bot does not store or process personal data, and any accidentally captured PII is automatically filtered out using heuristic rules documented in the repository.

⚙️ Rate Limiting Policy

Because the crawler is designed for large-scale data collection and may generate significant traffic, rate limiting is implemented to protect server resources. A threshold of 100 requests per 60 seconds is recommended for defensive blocking while still allowing legitimate crawling. The policy rationale is based on the bot’s documented crawl delay of 1 second; exceeding this rate likely indicates a misconfigured or abusive instance.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.