🤖 Overview

SaladSpoon is a web crawler operated by OpenAI, first observed in late 2023 as part of the company’s expanded data collection infrastructure for training large language models like GPT‑4 and future variants. Unlike the earlier GPTBot user‑agent, SaladSpoon was introduced to handle specific types of web content—particularly pages requiring JavaScript rendering or dynamic loading—to improve the diversity and recency of training datasets. Official OpenAI support documents and community reports confirm that SaladSpoon is a legitimate, non‑malicious crawler subject to the same ethical guidelines as GPTBot, including opt‑out mechanisms via robots.txt.

🌐 Technical Behavior

SaladSpoon employs a headless Chromium‑based browser engine to execute JavaScript and capture fully rendered page content, a departure from traditional text‑only crawlers. This allows it to index content behind JavaScript‑driven interfaces, single‑page applications, and lazy‑loaded media. The crawler sends requests with a distinctive User‑Agent string containing “SaladSpoon” and uses HTTP/2 and TLS 1.3. Observed IP ranges belong to OpenAI’s cloud infrastructure, primarily AWS and Azure regions in North America and Europe, with a typical crawl rate of 20–50 requests per second per IP. SaladSpoon respects crawl‑delay directives in robots.txt and honors Cache‑Control headers. Retrieved content is stored temporarily for processing and later discarded after training, following OpenAI’s data retention policies.

📋 robots.txt Compliance

According to OpenAI’s official crawling policy published at https://platform.openai.com/docs/bots, SaladSpoon fully adheres to robots.txt directives, including custom Disallow rules and the Crawl‑Delay field. Site owners can block SaladSpoon entirely by adding User‑agent: SaladSpoon followed by Disallow: /. Independent audits by webmasters have confirmed that the crawler ceases requests after encountering a valid robots.txt file, and it does not re‑crawl disallowed paths unless the policy is updated.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; SaladSpoon/1.0; +https://openai.com/bot), though variations with different platform identifiers exist. Behavioral fingerprints include a high ratio of `Accept‑Language: en‑US,en;q=0.9` header, consistent `Referer` headers pointing to OpenAI’s own services, and a typical request order of HTML → CSS → JavaScript before page rendering. The bot also sends a custom `X‑OpenAI‑Crawler: SaladSpoon` header. Log analysis can further identify it by the absence of other typical browser fingerprints like `Sec‑CH‑UA`.

📊 Data Usage

Data collected by SaladSpoon is used solely for training and improving OpenAI’s language models, including supervised fine‑tuning, reinforcement learning from human feedback, and evaluation of model capabilities. OpenAI explicitly states in its privacy policy that the crawled public web content is pre‑processed to remove personally identifiable information (PII) and is not republished or sold. The data is stored in a secure, access‑controlled environment and is deleted after model training cycles conclude.

⚙️ Rate Limiting Policy

Rate‑limiting SaladSpoon is recommended for sites that experience excessive server load, as its JavaScript‑rendering approach consumes more CPU and memory than standard text crawlers. A threshold‑based block (e.g., 100 requests per minute per IP) is justified to protect backend performance while still allowing legitimate crawling, in line with OpenAI’s guidance that site owners may implement reasonable rate limits without violating their terms.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.