🤖 Overview

takeout is a web crawler operated by Takeout Inc., a data aggregation company that collects restaurant menus, hours, and reviews from public websites to power their food discovery platform. The bot was first documented in late 2022 and adheres to standard web crawling practices, as described in their official documentation at https://takeout.com/crawler. Its purpose is strictly limited to gathering publicly available hospitality data.

🌐 Technical Behavior

The crawler uses a combination of HTTP/1.1 and HTTP/2 requests, typically sending a burst of 5-10 requests per minute per domain, with a default crawl delay of 30 seconds. It respects the Cache-Control and If-Modified-Since headers to reduce redundant bandwidth consumption. The user agent string is Takeout/1.0 and Mozilla/5.0 (compatible; Takeout/1.0; +https://takeout.com/bot). The bot primarily targets HTML pages and sitemaps, ignoring binary files such as images, PDFs, and videos unless explicitly linked via a nofollow-free path. IP addresses are drawn from the ranges 34.192.0.0/10 (Amazon Web Services, ASN 16509) and 35.190.0.0/16 (Google Cloud, ASN 15169), as published in their official IP disclosure list at https://takeout.com/ips. The crawler also respects X-Robots-Tag headers and supports robots.txt Crawl-delay directives.

📋 robots.txt Compliance

Takeout Inc. publicly states on their crawler information page that their bot fully honors robots.txt directives, including Disallow and Crawl-delay instructions. Evidence from their own robots.txt file at https://takeout.com/robots.txt confirms they include an explicit Disallow: /private/ rule for their own site, mirroring the standard they recommend external site operators adopt. However, they note in their policy that they may ignore Disallow for pages that are publicly accessible via direct links from other domains, a practice consistent with non-malicious aggregators.

🔍 Detection Indicators

The primary User-Agent string is Takeout/1.0 and Mozilla/5.0 (compatible; Takeout/1.0; +https://takeout.com/bot). Additional behavioral fingerprints include a consistent request interval of 30 seconds, a strong preference for text/html content type, and the use of Accept-Language: en-US header. The bot does not send custom HTTP headers beyond standard ones, and it respects ETag and Last-Modified headers. Its referrer field is usually empty or set to https://takeout.com.

📊 Data Usage

Collected data is used exclusively for the Takeout food discovery platform, including real-time menu aggregation, pricing updates, and review collection from publicly available sources. The data is not employed for AI training, resale, or third-party licensing, as per their privacy policy at https://takeout.com/privacy. Takeout Inc. also provides an opt-out form for site owners who wish to block all collection, with a 48-hour processing window.

⚙️ Rate Limiting Policy

While takeout is a legitimate bot, its crawling can be aggressive during initial indexation of a new site, potentially causing resource spikes despite its built-in delay. Web application operators should rate-limit at 10 requests per minute per IP to protect server performance while still enabling the bot to complete its data collection, as recommended by Takeout Inc.'s own guidelines published at https://takeout.com/rate-limits.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.