🤖 Overview

OpenAI.com refers to the suite of web crawlers operated by OpenAI, primarily GPTBot (announced August 2023) and OAI-SearchBot (released May 2024). GPTBot collects publicly accessible text, images, and metadata to train and improve OpenAI’s generative AI models, including GPT-4 and future iterations. OAI-SearchBot, by contrast, powers OpenAI’s real-time search capabilities (e.g., ChatGPT’s web search feature) and does not train models. Both crawlers are documented at platform.openai.com/docs/gptbot and platform.openai.com/docs/oai-searchbot.

🌐 Technical Behavior

GPTBot crawls using HTTP/1.1 and HTTP/2 with a default request rate of approximately 10–20 requests per second per IP, though it dynamically throttles if it receives 429 or 503 responses. It supports IPv4 and IPv6; OpenAI publishes its IP ranges in the JSON format at https://api.openai.com/v1/ip_ranges.json (prefixes include 23.98.0.0/16, 40.127.0.0/16, and 2603:1000::/28). The crawler respects Cache-Control, ETag, Last-Modified headers, and obeys robots.txt directives. GPTBot also sends a User-Agent token and an optional From header (used for debugging). OAI-SearchBot behaves similarly but maintains a separate user-agent string and distinct IP pools. Both crawlers advertise their presence via a robots.txt crawl-delay directive (default 1 second) if specified by the site owner.

📋 robots.txt Compliance

OpenAI explicitly states that GPTBot and OAI-SearchBot follow robots.txt rules as defined in the Robots Exclusion Protocol. On their official documentation pages (platform.openai.com/docs/gptbot#robots-txt) they confirm that if a site sets Disallow: / for GPTBot, no content from that path will be used for training. Similarly, OAI-SearchBot honors Disallow directives. OpenAI also respects noindex meta tags and X-Robots-Tag HTTP headers for both crawlers. There are no known incidents of systematic violation; OpenAI has a public policy to comply with site owners’ opt-out instructions.

🔍 Detection Indicators

User-Agent strings: GPTBot uses Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot). OAI-SearchBot uses Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/oai-searchbot). Behavioral fingerprints include a high frequency of HEAD requests before GET, a default Accept header of text/html,application/xhtml+xml, and a Connection header set to keep-alive. Both crawlers append a Via header indicating OpenAI’s proxy infrastructure. IP ranges are verified via the published JSON feed. Web servers can also check for the presence of the From header (optional) that may contain an internal identifier.

📊 Data Usage

Data collected by GPTBot is used exclusively for AI model training, including foundation model improvement, fine-tuning, and safety evaluation. OpenAI does not use OAI-SearchBot data for training; instead, it feeds OpenAI’s real-time search index to provide up-to-date answers in ChatGPT. Both crawlers exclude paywalled, personally identifiable, or adult content per their stated policies. Public web pages that are publicly accessible and not blocked via robots.txt or meta tags may be ingested. OpenAI provides a data-usage portal (platform.openai.com/docs/usage) where site owners can monitor crawl activity.

⚙️ Rate Limiting Policy

Site administrators are encouraged to rate-limit GPTBot and OAI-SearchBot to protect server resources because even though these crawlers are legitimate, they can generate sustained load exceeding 10 requests per second. A recommended threshold is 200 requests per minute per IP; returning 429 Too Many Requests prompts the crawler to back off. OpenAI’s own documentation advises rate-limiting as a standard defense against aggressive crawl patterns, noting that the bots will respect the response and reduce their rate. This policy balances the value of AI training data with the need to prevent resource exhaustion on smaller sites.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.