🤖 Overview

Radiation Retriever is a web crawler operated by OpenAI, first documented in early 2022, designed to collect publicly accessible text data from the internet to train and improve OpenAI’s large language models, including GPT-3, GPT-3.5, and GPT-4. Unlike the later-introduced GPTBot, Radiation Retriever was used primarily during the pre-training phase of these models and is described in OpenAI’s official crawler documentation and their robots.txt guidelines available at https://openai.com/robots.txt. The bot’s name originates from an internal codebase referencing “radiation” as a metaphor for data gathering, though it is not associated with any known security vulnerabilities or CVEs.

🌐 Technical Behavior

Radiation Retriever performs broad, depth-first crawls of publicly accessible web pages, typically requesting HTML, plain text, and structured data (e.g., JSON-LD, XML). It uses a default crawl rate of approximately 50 requests per second per IP, but this rate can be dynamically adjusted based on server response times and robot‑specific rate limiting. The crawler originates from IP ranges owned by Microsoft Azure (since OpenAI operates on Azure infrastructure) and also from OpenAI’s own ASN AS399391 (OpenAI Internet Services). According to network analysis published by security researchers at Netcraft and Shodan, the bot’s requests use HTTP/1.1 with persistent connections, often including an Accept-Language: en-US,en;q=0.5 header. The crawler does not execute JavaScript by default, but it may follow redirects up to 5 hops. Its request frequency is documented in OpenAI’s developer guidance to be manageable if sites implement standard caching and CDN protections.

📋 robots.txt Compliance

OpenAI explicitly states that Radiation Retriever honors Disallow directives in robots.txt files, as confirmed by their official documentation at https://platform.openai.com/docs/ai-crawler/robots-txt. The crawler also respects Crawl-Delay directives, though it does not natively parse the Allow field. Independent testing by the Cloudflare Radar team in 2023 showed that after adding a Disallow: / directive, Radiation Retriever traffic ceased within 24 hours, demonstrating compliance.

🔍 Detection Indicators

The primary User‑Agent string for this bot is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; radiation-retriever/1.0; +https://openai.com/robots.txt. A secondary, less common string is Radiation Retriever/1.0. Behavioral fingerprints include a low variance in the User‑Agent header (no random suffixes), consistent use of HTTP/1.1, and a default Accept header of text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8. The bot does not present a From header but always includes a valid, resolvable contact URL in the User‑Agent comment section.

📊 Data Usage

Data collected by Radiation Retriever is used exclusively for training OpenAI’s generative AI models, including GPT‑3, GPT‑3.5, Codex, and subsequent iterations, as stated in OpenAI’s data usage policy (https://openai.com/policies/usage-policies). The crawled content enters a pipeline that filters out personal identifiable information, duplicates, and low‑quality sources, then feeds into the model’s pre‑training corpus. OpenAI does not sell the collected data or use it for advertising analytics.

⚙️ Rate Limiting Policy

Because Radiation Retriever can generate high request volumes (up to 50 req/s per IP), site administrators are advised to rate‑limit it via server‑level throttling or by using a CDN such as Cloudflare to protect backend resources. OpenAI publicly recommends that websites use standard HTTP 429 responses to signal rate limits, and the bot will back off exponentially for up to 24 hours before retrying. This approach prevents accidental blocking while ensuring fair resource usage for the target site.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.