🤖 Overview

Applebot is the web crawling agent developed and operated by Apple Inc., first publicly documented in 2015. Its primary purpose is to index web content for Apple’s search services, including Siri, Spotlight Suggestions, and the Applebot-Extended data usage control introduced in 2024 for AI training opt-out. Applebot also feeds into Apple’s foundation models used in features like Intelligence in iOS, iPadOS, and macOS. According to Apple’s official support documentation, the crawler respects the robots.txt file and the Applebot-Extended directive, which allows site owners to exclude content from being used for AI model training while still being indexed for search.

🌐 Technical Behavior

Applebot performs both HTTP/1.1 and HTTP/2 requests, with a default crawl frequency that adapts to site responsiveness and server load. It uses a Dynamic IP range belonging to Apple’s ASN (AS714, AS2709, AS6185), primarily originating from the 17.0.0.0/8 block. Applebot supports gzip and deflate compression and sends a User-Agent string that includes the main product token and version information. The crawler follows standard crawl patterns, fetching robots.txt at the start of each session and respecting Crawl-Delay directives. Applebot also uses the Applebot-Extended token to allow fine-grained control over data usage.

📋 robots.txt Compliance

Applebot fully complies with the Robots Exclusion Protocol, including the Disallow directive and the Crawl-Delay setting. Apple’s official guidance states that Applebot respects both standard robots.txt rules and the proprietary Applebot-Extended token, which site owners can use to block content from being used for Apple’s AI training while still allowing indexing. Evidence from Apple’s developer site confirms that the crawler reads and obeys all directives, including those at the per-path and per-rule level.

🔍 Detection Indicators

The primary User-Agent string is “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15 (Applebot/0.1)” for desktop crawls and “Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1 (Applebot/0.1)” for mobile. Additionally, the Applebot-Extended user-agent token can be used in robots.txt for AI training control.

📊 Data Usage

Collected data is used primarily for search indexing in Apple’s services such as Safari search, Siri, and Spotlight. As of 2024, Apple also uses publicly crawled content to train its foundational AI models, including those powering Apple Intelligence features. Apple provides a distinct opt-out mechanism (Applebot-Extended) for site owners who wish to prevent their data from being used in AI training while allowing search indexing.

⚙️ Rate Limiting Policy

Applebot is rate-limited because its adaptive crawl frequency can still overwhelm small or poorly optimized servers, especially during initial deep crawls. Site owners are advised to implement threshold-based blocking (e.g., via mod_evasive or similar tools) at a generous buffer above Applebot’s typical request rate, which averages around 50 requests per second per IP, to protect server resources without outright blocking a legitimate search and AI training crawler.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.