🤖 Overview

ChatGPT Agent is an AI-powered web crawler operated by OpenAI, introduced in early 2024 as part of the ChatGPT ecosystem. Its primary purpose is to browse the web in real time on behalf of users who enable the “Browse with Bing” feature within the ChatGPT interface, fetching up‑to‑date information to augment model responses. The agent feeds retrieved content directly into the conversational context, not into a separate training dataset, per OpenAI’s official documentation at help.openai.com.

🌐 Technical Behavior

The agent uses a headless Chromium browser to render pages, executing JavaScript and handling redirects similarly to a human user. It makes requests from OpenAI’s validated IP ranges (published at openai.com/trust), typically originating from US-based data centers. Request frequency is moderate—on the order of a few requests per minute per user session—but can surge during peak usage. The crawler follows HTTP 3xx redirects and respects Cache‑Control headers; it does not intentionally bypass content behind login walls. OpenAI reports the agent uses the same rate‑limiting infrastructure as its API, with per‑IP throttling to prevent overwhelming origin servers.

📋 robots.txt Compliance

OpenAI explicitly states that the ChatGPT Agent obeys robots.txt directives, including both generic Disallow rules and those targeting the User‑Agent token “ChatGPT‑User”. Official documentation (platform.openai.com/docs/plugins/browsing) confirms the agent checks robots.txt before each request and will refuse to fetch pages blocked by site owners.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; ChatGPT‑User/1.0; +https://openai.com/chatgpt‑user). Additional fingerprints include a distinctive Accept‑Language header of “en‑US,en;q=0.9” and a Sec‑CH‑UA header indicating a Chromium‑based engine. The agent’s IP addresses belong to OpenAI’s ASN 396982 (AS‑OPENAI) and are listed in the public ip‑ranges.json file at openai.com.

📊 Data Usage

Collected content is used solely for real‑time answer generation within the current ChatGPT conversation. OpenAI does not retain the fetched page text beyond the user’s session for training purposes, as per their privacy policy. The data is discarded after the session ends unless the user manually saves the conversation.

⚙️ Rate Limiting Policy

Because the agent can make repeated requests across many simultaneous users, it is rate‑limited to protect server resources and prevent service degradation. Threshold‑based blocking (e.g., more than 10 requests per second from a given IP) is justified under standard API rate‑limiting practices, ensuring fair access for all websites.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.