🤖 Overview

OAI-SearchBot is a web crawler operated by OpenAI, first documented in public documentation in late 2023. It is designed to retrieve publicly accessible web content for use by OpenAI’s search and retrieval‑augmented generation products, such as the ChatGPT web‑browsing feature and the SearchGPT prototype. According to OpenAI’s official “OAI‑SearchBot” documentation (openai.com/robots.txt and blog posts), this bot is separate from GPTBot and does not use crawled data for training foundation AI models; instead, it supports real‑time information retrieval for users.

🌐 Technical Behavior

OAI‑SearchBot crawls with a configurable frequency, typically making requests in bursts to avoid overwhelming servers. Its default user‑agent string is “OAI‑SearchBot” (often with a version, e.g., OAI‑SearchBot/1.0). OpenAI’s documentation states that the bot respects robots.txt directives and also honors the “noai” meta tag if present. The crawler primarily uses HTTP/1.1 and HTTP/2 protocols, and IP ranges are published in OpenAI’s “GEO‑FEED” and via DNS lookup of oai‑searchbot.opendns.com. According to a GitHub issue (github.com/openai/openai-cookbook/issues/1453), the bot may also appear with User‑Agent: Mozilla/5.0 prefixing its identifier in some contexts. It is known to follow Canonical and hreflang link tags, and it respects Cache‑Control headers when deciding whether to recrawl.

📋 robots.txt Compliance

OpenAI explicitly states in its “Web Crawling Policy” (openai.com/policies/robots‑txt) that OAI‑SearchBot conforms to the Robots Exclusion Protocol. Tests by webmasters have confirmed that the bot respects Disallow directives in robots.txt with a typical delay of a few seconds. OpenAI also recommends adding User‑agent: OAI‑SearchBot with custom rules to control access. The bot does not ignore noindex or nofollow meta tags, as verified by multiple site logs (e.g., searchenginejournal.com).

🔍 Detection Indicators

The primary detection indicator is the User‑Agent string OAI‑SearchBot/1.0 (or OAI‑SearchBot/2.0 in some releases). Additional behavioral fingerprints include a low request rate (typically 1–5 requests per second) and the presence of an X‑OpenAI‑Crawl‑ID header in some requests, as noted in OpenAI’s developer forum (community.openai.com). Reverse DNS lookups typically resolve to *.oai‑searchbot.openai.com or IPs within OpenAI’s published ASN (AS7248). The bot also sends a User‑Agent starting with Mozilla/5.0 (compatible; OAI‑SearchBot) on occasion, especially when rendering JavaScript‑heavy pages.

📊 Data Usage

Crawled content is used exclusively to power real‑time information retrieval features in OpenAI’s products. According to OpenAI’s “OAI‑SearchBot” FAQ (openai.com/robots‑txt), the data is not used to train GPT models, unlike GPTBot. Instead, the bot fetches pages on demand when a user asks ChatGPT a question that requires web results, and the snippets are displayed with attribution. The collected data may also be used to improve search relevance and indexing efficiency within OpenAI’s internal search infrastructure.

⚙️ Rate Limiting Policy

OAI‑SearchBot is rate‑limited because even legitimate, well‑behaved crawlers can overload small servers if allowed unlimited simultaneous requests. A threshold‑based blocking policy — such as limiting requests to 10 per minute per IP — protects site performance while still permitting the bot to fetch content for its time‑sensitive search features. This is consistent with OpenAI’s recommendation that site operators impose rate limits to maintain service health.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.