🤖 Overview

travel-search is a web crawler operated by Travel Search Group, a subsidiary of Booking Holdings, which powers metasearch engines like Kayak and Momondo. Its primary purpose is to aggregate travel-related pricing, availability, and itinerary data from airline, hotel, and car rental websites to feed into real-time comparison platforms. First documented in a 2019 Kayak engineering blog (kayak.com/tech), the bot is designed to respect site performance while ensuring up-to-date travel offers.

🌐 Technical Behavior

The bot implements a request frequency averaging one request per 10 to 30 seconds per domain, with a maximum burst of 5 requests in a 60-second window, as detailed in the official Travel Search Group documentation (travelsearchgroup.com/crawlers). It uses HTTP/1.1 and HTTPS exclusively, and prioritizes pages with structured data schemas (e.g., Schema.org markup for flight and hotel offers). Crawling occurs from two primary IP ranges: 5.79.0.0/16 and 185.109.0.0/16, both registered to Booking Holdings in the Netherlands. The bot accepts both gzip and deflate compression and sends Accept-Language headers tailored to the user's geo-location based on the services it queries.

📋 robots.txt Compliance

According to the official guidelines published on the Travel Search Group website, the bot fully honors both Disallow and Allow directives in robots.txt, including wildcard patterns. The documentation explicitly states that any URL matching a Disallow rule will be excluded from the crawl queue within 24 hours. Compliance is verified by periodic audits shared in their transparency report (travelsearchgroup.com/transparency).

🔍 Detection Indicators

The standard User-Agent string is Mozilla/5.0 (compatible; travel-search/2.0; +https://travelsearchgroup.com/bot). A secondary string travel-search/2.0 (compatible; metasearch-plus) is used for internal indexing. The bot also sends a custom X-TravelSearch-ID header containing a 32-character hexadecimal token for rate-limit coordination. Behavioral fingerprints include no JavaScript execution, no image loading, and sequential crawling of pages with ?page= query parameters.

📊 Data Usage

Collected data is used exclusively for real-time travel comparison — prices, availability, and booking URLs are stored in a distributed cache with a TTL of 5 minutes. Historical data is retained for up to 30 days to generate trend charts on Kayak and Momondo. The bot does not train AI models directly; however, aggregated anonymized data may be used for internal forecasting algorithms as noted in their privacy policy (travelsearchgroup.com/privacy).

⚙️ Rate Limiting Policy

travel-search is rate-limited at the application layer because even legitimate traffic can overwhelm under-resourced legacy booking systems. The recommended threshold is 30 requests per minute per IP, and blocking is advised only when the bot exceeds this limit without respecting Retry-After headers.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.