🤖 Overview

GPTBot is a web crawler operated by OpenAI, first publicly documented in August 2023 via their official blog and IP ranges published at https://openai.com/gptbot. Its primary purpose is to collect publicly accessible web content to train and improve OpenAI’s GPT-series language models, including GPT-4 and subsequent iterations. Unlike general search engine crawlers, GPTBot focuses on textual data suitable for natural language understanding and generation.

🌐 Technical Behavior

GPTBot uses HTTP/1.1 and HTTP/2 protocols, sending requests with a User-Agent string of Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.0; +https://openai.com/gptbot. It respects the robots.txt standard and checks the X-Robots-Tag HTTP header for per-page directives. Crawl frequency is moderate, typically one request per second per domain, but can burst up to 10 requests in short intervals. OpenAI publishes its crawler IP ranges in CIDR form (e.g., 20.190.128.0/18, 40.80.0.0/13) which are resolvable via the openai.com/_static/gptbot-ips.txt endpoint. GPTBot does not execute JavaScript or submit forms; it only fetches static HTML and associated resources like CSS and images when relevant for context understanding.

📋 robots.txt Compliance

OpenAI explicitly states that GPTBot respects robots.txt instructions, including both Disallow and Allow directives. In August 2023, OpenAI added a dedicated User-agent: GPTBot entry to their own robots.txt as an example, and encourages site owners to block GPTBot using that token. Third-party testing by security researchers (e.g., @thezedwards on X) confirmed that GPTBot ceases crawling when a Disallow: / rule is present. However, OpenAI notes that content blocked via robots.txt is not used for training, but content accessible to other bots may still be collected if not explicitly excluded for GPTBot.

🔍 Detection Indicators

The primary identifier is the User-Agent string Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.0. Additional indicators include the From header (rarely sent) and a custom X-GPTBot header sometimes observed in logs. Reverse DNS lookups on request IPs resolve to *.openai.com subdomains, such as gptbot-pool-*.openai.com. The bot also sends a Accept: text/html,application/xhtml+xml header and does not include a Referer field. Rate‑limiting rules can be applied using the User-agent token GPTBot in both robots.txt and .htaccess.

📊 Data Usage

Data collected by GPTBot is used exclusively for training OpenAI’s large language models. According to OpenAI’s privacy policy and the GPTBot documentation, content is processed, filtered, and stripped of personally identifiable information (PII) before inclusion in training datasets. The resulting models are then released through APIs and products like ChatGPT and the GPT‑4 family. OpenAI also uses the data to improve model safety, factuality, and adherence to instructions, but does not share raw crawled content publicly.

⚙️ Rate Limiting Policy

Although GPTBot is not malicious, its aggressive collection speed can degrade server performance for shared hosting environments or sites with limited bandwidth. Rate limiting is justified because the bot does not throttle itself below one request per second per domain, and without limits, a burst of 10+ requests can trigger resource exhaustion. Threshold-based blocking (e.g., 50 requests per minute from a single GPTBot IP) is a standard precaution to protect application availability without permanently denying access.

Googlebot;

🤖 Overview

Googlebot is the web crawling component of Google Search, operated by Alphabet Inc. It has been active since 1998 and is documented at https://developers.google.com/search/docs/crawling-indexing/googlebot. Its core purpose is to discover and index web pages for Google’s search engine, ranking algorithms, and features like Google News and Google Discover. Googlebot also powers Googlebot Image, Googlebot Video, and Googlebot News variants for specialized content types.

🌐 Technical Behavior

Googlebot uses HTTP/2 and HTTP/3 protocols and employs a two‑phase crawl: first a mobile‑first request using a smartphone User‑Agent (e.g., Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.230 Mobile Safari/537.36), followed by a desktop variant. It supports rendering JavaScript via a headless Chrome instance (Web Rendering Service) and may fetch CSS, images, and JavaScript files needed for page rendering. Googlebot’s crawling frequency is adaptive; it can send hundreds of requests per minute to popular sites but respects the Crawl‑Delay directive in robots.txt. The bot originates from IP ranges published in Google’s SPF records and verified via reverse DNS to *.googlebot.com. It also uses HTTP/2 Server Push and respects Cache‑Control headers. Googlebot sends a User‑agent: Googlebot/2.1 (+http://www.google.com/bot.html) and includes a Via header for proxy detection.

📋 robots.txt Compliance

Googlebot is one of the most compliant crawlers; it strictly follows robots.txt rules, including the Disallow, Allow, Crawl‑Delay, and Sitemap directives. It also honors X‑Robots‑Tag and noindex/nofollow meta tags. Google’s official documentation states that ignoring robots.txt can lead to de‑indexing or manual penalties. However, Googlebot may ignore Disallow for URLs it deems essential for security (e.g., re‑crawling after site‑wide changes) but such behavior is rare and documented. The Googlebot Smartphone and Googlebot Desktop variants each use the same robots‑token Googlebot, so a single rule applies to both.

🔍 Detection Indicators

Primary identifiers: User‑Agent strings (smartphone and desktop variants above, plus Googlebot‑Image/1.0 for images). Reverse DNS resolves to crawl‑xxx‑xxx‑xxx‑xxx.googlebot.com. Forward DNS confirmation is recommended: perform a PTR lookup on the IP, then an A‑record query on the returned domain – it must resolve back to the original IP. Googlebot also uses a X‑Forwarded‑For header when behind proxies, and typically lacks a Referer field. The Accept‑Language header is often set to en‑US,en;q=0.9. Googlebot does not set DNT (Do Not Track) headers. Log analysis can detect bursts of requests from the same IP range with identical User‑Agent and no other variations.

📊 Data Usage

Data collected by Googlebot feeds Google’s search index, used to build the search result ranking system (PageRank, BERT, MUM). Crawled content is also used for AI‑powered features like Google SGE (Search Generative Experience), snippet generation, and entity extraction. Google does not use crawled data for training general‑purpose AI models (e.g., LaMDA) without explicit consent, as per their updated AI training policy in July 2023. The data is stored in massive distributed databases (Bigtable, Spanner) and periodically re‑crawled to detect changes.

⚙️ Rate Limiting Policy

Googlebot’s aggressive adaptive crawling can overwhelm small sites or those with poor caching; it may send 100+ requests within seconds during initial discovery. Rate limiting is recommended because Googlebot does not inherently throttle based on server load—it responds only to Crawl‑Delay directives or HTTP 429 responses. A threshold of 200 requests per minute per IP from a Googlebot range is a reasonable starting point to protect application availability while still allowing indexing. Using .htaccess or a WAF to apply a 20‑request‑per‑second cap for Googlebot IPs is a common practice.

Bingbot;

🤖 Overview

Bingbot is the web crawler for Microsoft Bing, operated by Microsoft Corporation. It was launched in 2010 to replace the earlier msnbot and is documented at https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0. Its primary purpose is to discover and index web pages for Microsoft’s search engine, which also powers Microsoft Edge’s search and Copilot’s web grounding features. Bingbot also supports image, video, and news crawling through specific variants.

🌐 Technical Behavior

Bingbot uses HTTP/1.1 and HTTP/2 protocols and sends requests with a User‑Agent string of Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bingbot/2.0; +http://www.bing.com/bingbot.htm). It also uses a msnbot‑media/1.1 variant for images and BingPreview/1.0b for social snippets. Bingbot supports JavaScript rendering (using a headless browser) and fetches linked resources like CSS, JavaScript, and images. Crawl frequency is high but respects Crawl‑Delay in robots.txt. Microsoft publishes Bingbot IP ranges in a regularly updated file at https://www.bing.com/toolbox/bingbot.json. The bot originates from IPs belonging to Microsoft Azure and Microsoft’s own ASN (AS8075). It checks both robots.txt and X‑Robots‑Tag headers, and also respects noindex, nofollow meta tags. Bingbot uses a From header (rarely) and maintains a persistent TCP connection for multiple requests per URL.

📋 robots.txt Compliance

Bingbot fully respects robots.txt, including Disallow, Allow, and Crawl‑Delay directives. Microsoft’s documentation states that ignoring robots.txt will result in de‑indexing. The User‑agent: bingbot token is used for both the main crawler and its media variants. Bingbot also honors the X‑Robots‑Tag and noindex meta tags. However, it may ignore Disallow if the URL is part of a sitemap that explicitly allows it (a known edge case).

🔍 Detection Indicators

Main User‑Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm). Additional strings include msnbot/1.1 (+http://search.msn.com/msnbot.htm) for legacy compatibility and BingPreview/1.0b for snippet generation. Reverse DNS resolves to *.search.msn.com or *.msnbot.com. Forward DNS validation is recommended. Bingbot includes a From: [email protected] header inconsistently. It also sends Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8. Burst patterns: Bingbot may send 10–20 requests in rapid succession from different IPs within the same /24 range, then pause for several seconds.

📊 Data Usage

Data collected by Bingbot is used to build Microsoft Bing’s search index, which powers search results, snippets, and the Microsoft Copilot AI assistant’s web grounding. Microsoft also uses crawled content for training its AI models (e.g., GPT‑4 via Azure OpenAI) but only after filtering personal information and respecting opt‑out signals. The data is stored in Azure Storage and processed by indexing pipelines that extract text, metadata, and relationships for ranking algorithms like RankNet and BERT based models.

⚙️ Rate Limiting Policy

Bingbot can be aggressive—especially during first‑time crawls of new domains, where it may issue up to 30 requests per second from multiple IPs. Rate limiting is recommended because without it, the bot can cause server load spikes that impact legitimate users. A threshold of 100 requests per minute per Bingbot IP or using a Crawl‑Delay: 2 directive in robots.txt is standard practice to balance indexing needs with site performance. Blocking beyond thresholds (e.g., 429 responses after 150 requests) is acceptable to protect application availability.

Applebot;

🤖 Overview

Applebot is the web crawler operated by Apple Inc., first introduced in 2015 and documented at https://support.apple.com/en-us/HT203609. It is used by Apple’s search technologies, including Siri, Spotlight suggestions, and Apple News, as well as for training Apple’s AI models (e.g., foundation models in iOS). Unlike general search engine bots, Applebot’s focus is on generating high‑quality, privacy‑preserving search results and AI knowledge bases for Apple’s ecosystem.

🌐 Technical Behavior

Applebot uses HTTP/2 and supports both IPv4 and IPv6. It sends requests with User‑Agent strings: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15 (Applebot/0.1) for desktop and Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/602.1 (Applebot/0.1) for mobile. Apple publishes its IP ranges at https://support.apple.com/en-us/HT204683 (list of CIDR blocks like 17.0.0.0/8). Crawl frequency is moderate (1–5 requests per second per IP) and the bot respects Crawl‑Delay in robots.txt. Applebot also checks X‑Robots‑Tag and noindex meta tags. It does not execute JavaScript for indexing but does fetch page HTML and associated resources. Applebot uses persistent connections with keep‑alive for 15–60 seconds.

📋 robots.txt Compliance

Applebot fully respects robots.txt, including the User‑agent: Applebot token. Apple’s documentation explicitly states that it honors Disallow and Allow directives, and also obeys Crawl‑Delay when specified. The bot also interprets the X‑Robots‑Tag header. However, Apple notes that content blocked via robots.txt may still be used for “basic page discovery” if referenced from an allowed page, but that content is not indexed or used for training.

🔍 Detection Indicators

Primary User‑Agent strings: the Safari‑based variants above, and a simpler Applebot token (only used in robots.txt). Reverse DNS resolves to *.applebot.apple.com. Forward DNS validation: IP → PTR → A record must match. Applebot also includes a From: [email protected] header consistently. It lacks a Referer header and typically sends Accept‑Language: en‑US,en;q=0.9. Applebot’s request size is small (no large payloads) and it uses Connection: keep‑alive. It can be differentiated from Safari browsers by the presence of the Applebot token in the User‑Agent and the From header.

📊 Data Usage

Data collected by Applebot is used to improve Apple’s search experiences in iOS, macOS, and Siri, as well as for training Apple’s AI models (e.g., the on‑device foundation models that power predictive text and photo search). Apple states that crawled data is anonymized and aggregated, with strict privacy controls: no IP addresses are stored long‑term, and data is retained only as long as necessary for model training. The data is not used for profiling or advertising.

⚙️ Rate Limiting Policy

Applebot’s request rate is generally modest, but during a re‑crawl of a high‑traffic site, it can ramp up to 10 requests per second from a single IP. Rate limiting is recommended because without a Crawl‑Delay directive, Applebot will saturate a small server’s available connections. A threshold of 60 requests per minute per Applebot IP is a balanced approach, preserving indexing quality while protecting application resources. Using a 429 Too Many Requests response after exceeding this limit is standard and accepted by Apple.

DuckDuckBot;

🤖 Overview

DuckDuckBot is the web crawler for DuckDuckGo, a privacy‑focused search engine operated by DuckDuckGo, Inc. It was first introduced in 2011 and is documented at https://duckduckgo.com/duckduckbot. DuckDuckBot’s purpose is to index web pages for the search engine’s results, which emphasize user anonymity and do not track browsing history. The bot also powers DuckDuckGo’s Instant Answers and !bang commands, but does not feed data into AI training models like ChatGPT.

🌐 Technical Behavior

DuckDuckBot uses HTTP/1.1 primarily, with occasional HTTP/2 for larger downloads. It sends requests with a User‑Agent of Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; +https://duckduckgo.com/duckduckbot). There is also a non‑HTTPS variant: DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html). DuckDuckBot does not execute JavaScript or fetch CSS; it only downloads raw HTML and text to minimize user‑data collection. Crawl frequency is low to moderate (one request per second per domain on average). DuckDuckGo publishes its crawler IP ranges in the duckduckgo‑crawler‑ips.txt file hosted at https://duckduckgo.com/duckduckbot. The IPs belong to AS53667 (DuckDuckGo). DuckDuckBot respects Crawl‑Delay in robots.txt and also checks the X‑Robots‑Tag header. It does not use persistent connections and closes TCP after each request.

📋 robots.txt Compliance

DuckDuckBot fully respects robots.txt, including Disallow, Allow, and Crawl‑Delay. The User‑agent: DuckDuckBot token is used for both HTTPS and HTTP variants. DuckDuckGo states that they honor noindex meta tags as well. The bot will also follow sitemap directives if included. There are no known cases of DuckDuckBot ignoring robots.txt rules.

🔍 Detection Indicators

User‑Agent strings: as above – DuckDuckBot-Https/1.1 and DuckDuckBot/1.1. The bot also sends a From: [email protected] header consistently. Reverse DNS resolves to *.duckduckgo.com. Forward DNS validation is straightforward: PTR lookup gives host‑xxx‑xxx‑xxx‑xxx.duckduckgo.com. DuckDuckBot includes an Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 header but no Accept‑Encoding (though it does accept gzip). It lacks a Referer field and sends Connection: close. Its requests are very lightweight (no JavaScript, minimal headers).

📊 Data Usage

Data collected by DuckDuckBot is used exclusively for DuckDuckGo’s search index. DuckDuckGo does not train AI models on user search queries or crawled content, in line with their privacy policy. The data is stored in anonymized form; raw page content is processed into an inverted index and then discarded. DuckDuckGo uses the index for organic search results and for its Zero‑Click Info boxes. No personal data is retained, and the bot does not log IP addresses beyond transient request handling.

⚙️ Rate Limiting Policy

DuckDuckBot’s crawl rate is low (around 1 request per second), making it one of the politest major crawlers. Rate limiting is generally unnecessary, but in rare cases a misconfigured site may trigger a burst. A threshold of 30 requests per minute from a single DuckDuckBot IP is more than sufficient to protect server resources. The rationale for rate limiting is to prevent accidental resource exhaustion during re‑crawls of very large sites; DuckDuckGo officially supports using Crawl‑Delay in robots.txt to control its speed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.