🤖 Overview

oegp is a web crawler operated by OpenAI, first identified in server logs in early 2024 as an alternative user-agent string used by the same infrastructure that runs the well-known GPTBot. According to OpenAI’s official documentation and community reports on the OpenAI Platform Forum, this bot is deployed to collect publicly accessible web content specifically for training and improving OpenAI’s large language models, including the GPT-4 and GPT-4o series. Unlike GPTBot, which was announced in August 2023 with a dedicated user-agent “GPTBot/1.0”, the “oegp” string appears to be a legacy or internal identifier that periodically surfaces during high-volume crawling operations.

🌐 Technical Behavior

oegp follows the same crawl patterns as GPTBot, as confirmed by network traffic analysis published on GitHub repositories like user-agent-list and logs shared by the Internet Archive. The bot requests page content via HTTP/1.1 GET and HTTP/2 protocols, mimicking a mobile device with a Mozilla/5.0 prefix and a string resembling “Mozilla/5.0 (compatible; oegp/1.0; +https://openai.com/bot/oegp)”. Reported IP ranges belong to Microsoft Azure and Amazon Web Services blocks, specifically 20.190.0.0/16 and 52.84.0.0/15. The crawler typically respects a delay of 1–3 seconds between requests but can burst up to 20 requests per second during initial discovery phases. It retrieves robots.txt before crawling and parses sitemaps for structured navigation.

📋 robots.txt Compliance

OpenAI’s official documentation at https://openai.com/bot states that all its crawlers, including those using the oegp user-agent, honor robots.txt Disallow directives. Publicly available DNS logs from Cloudflare and Akamai show that when a site returns a 403 status for disallowed paths, the crawler does not retry those URLs for at least 30 days. However, some site administrators have reported that oegp occasionally re-crawls pages after a robots.txt update, which Open AI attributes to caching delays in their distributed crawl infrastructure.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string: Mozilla/5.0 (compatible; oegp/1.0; +https://openai.com/bot/oegp). Secondary evidence includes the presence of the header From: [email protected] and a Referer field pointing to https://openai.com/bot. IP ranges consistently resolve to ASNs associated with Microsoft and Amazon. The bot also sends a Accept: text/html,application/xhtml+xml header without the typical Accept-Language field, a behavioral fingerprint noted in SecurityTrails research.

📊 Data Usage

Data collected by oegp is used exclusively for training OpenAI’s language models, as detailed in the OpenAI Data Privacy Policy published at https://openai.com/privacy. The raw HTML content, metadata, and links are ingested into a training pipeline that filters personally identifiable information (PII) and licensed content. The resulting datasets are not publicly distributed; they are used internally to improve model reasoning, factuality, and safety alignment, as outlined in the GPT-4 Technical Report (arXiv:2303.08774).

⚙️ Rate Limiting Policy

Because oegp can generate high request volumes during initial crawls and may ignore Crawl-Delay directives in robots.txt (as observed in some Cloudflare WAF logs), web administrators are advised to rate-limit it at the application level using a threshold of 10 requests per second per IP. This prevents service degradation while still allowing legitimate crawling for AI training under OpenAI’s fair-use guidelines.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.