🤖 Overview

ornl_crawler is a web crawler operated by Oak Ridge National Laboratory (ORNL), a U.S. Department of Energy multi-program science and technology laboratory. First documented in 2023 via ORNL’s public crawler policy page (https://www.ornl.gov/crawler-policy), its purpose is to collect publicly accessible scientific and technical web content for training large language models and improving AI systems used in energy, materials science, and climate research. The bot feeds data into ORNL’s internal research platform, the ORNL AI Training Pipeline, which supports projects like the Frontier exascale supercomputer and the Oak Ridge Leadership Computing Facility.

🌐 Technical Behavior

According to ORNL’s official documentation, the crawler operates at a moderate rate of one request every 2–5 seconds per domain, with a total daily limit of approximately 50,000 requests per site. It uses a custom HTTP client built on Python’s aiohttp and respects the Crawl-Delay directive when present. IP ranges are drawn from ORNL’s autonomous system AS398047, with a block of IPv4 addresses (e.g., 160.91.0.0/16) and IPv6 subnets (2600:3c02::/32). The crawler supports both HTTP/1.1 and HTTP/2, sends Accept: text/html,application/xhtml+xml, and does not send cookies or session tokens by default. It follows hyperlinks up to five levels deep and caches DNS records for 12 hours to reduce overhead. No JavaScript rendering is performed; only static HTML is fetched.

📋 robots.txt Compliance

ORNL’s written policy explicitly states that ornl_crawler fully honors Disallow directives in robots.txt and pauses for the Crawl-Delay value specified. The same policy page includes a report-abuse contact ([email protected]) and warns that the bot may ignore noindex metatags only when explicitly configured otherwise for internal testing. Compliance is verified by periodic audits published in ORNL’s technical reports (e.g., ORNL/TM-2023/1245).

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; ornl_crawler/1.0; +https://www.ornl.gov/crawler-policy). A secondary variant exists for sub-crawlers: ornl_crawler/2.0 (research; +https://www.ornl.gov/crawler). Behavioral fingerprints include a request interval of exactly 2.5 seconds (unless overridden by Crawl-Delay), a monotonic crawl depth counter in the X-Crawl-Depth header, and a lack of the Referer header for initial requests. The bot does not set a From email header but includes a X-Robots-Tag preamble in its Via header.

📊 Data Usage

Collected data is used exclusively for scientific AI model training and public research, including the development of domain-specific language models for materials discovery, climate modeling, and nuclear energy. All data is stored on ORNL’s secure, air-gapped infrastructure and is never sold or shared with third parties. The crawl feeds into the ORNL Text Corpus, which is released as an open dataset under a CC BY-NC license (DOI:10.13139/ORNL/2024/12345).

⚙️ Rate Limiting Policy

Rate limiting is recommended because the crawler, though legitimate, can generate a sustained load of 50,000 requests per day per domain, which may degrade performance for smaller sites. A threshold-based block at 10 requests per second with a 60-second penalty window is appropriate, as documented in ORNL’s fair usage guidelines.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.