🤖 Overview

tygoprowler is a legitimate web crawler operated by Tygo, a Dutch energy comparison and consumer services platform based in Almere, Netherlands. Its primary purpose is to collect publicly available pricing data, energy tariff information, and product details from utility and service provider websites to feed into Tygo’s comparison engine, allowing users to find the best energy contracts, internet plans, and insurance offers. The bot was first documented in early 2022 and is explicitly listed in Tygo’s official documentation as a necessary tool for maintaining up-to-date comparison data.

🌐 Technical Behavior

tygoprowler typically sends HTTP GET requests at intervals of 2 to 5 seconds per host, with a maximum of 20 concurrent requests per domain, as stated in Tygo’s crawler policy page (available at tygo.nl/crawler). The bot uses IPv4 addresses from a dedicated range assigned to Tygo’s infrastructure, primarily 185.53.56.0/24, which are verifiable via reverse DNS lookups on dig results published by Tygo. It enforces a robots.txt cache of 24 hours and respects If-Modified-Since headers to avoid re-crawling unchanged pages. Crawl depth is limited to three levels by default, and the bot does not follow redirects to external domains unless explicitly allowed in the page’s meta tags.

📋 robots.txt Compliance

According to Tygo’s official developer blog (blog.tygo.nl/crawler-guidelines-2023), tygoprowler fully honors Disallow directives in robots.txt, including wildcard patterns and explicit path exclusions. The crawler also respects Crawl-Delay directives, with a default delay of 5 seconds if not specified. Tygo states that non-compliance reports are handled within 48 hours, and they provide a contact email ([email protected]) for webmasters to report issues.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; tygoprowler/2.0; +https://tygo.nl/bot), as confirmed in multiple web server logs and the official Tygo GitHub repository (github.com/tygo/tygoprowler). Secondary identifiers include a consistent X-Tygo-Crawler header set to true and a Referer header pointing to https://tygo.nl/. Behavioral fingerprints show the bot only requests HTML pages (ignoring .css, .js, image files) and includes a Accept-Language header of en,NL.

📊 Data Usage

Collected data — such as energy price per kWh, subscription fees, contract durations, and availability regions — is stored in Tygo’s database and used exclusively for consumer comparison results displayed on tygo.nl. No data is sold or used for AI training; Tygo’s privacy policy (tygo.nl/privacy) explicitly states that raw crawled data is deleted after 30 days and only aggregated pricing trends are retained for analytics. The bot does not collect personally identifiable information.

⚙️ Rate Limiting Policy

Rate-limiting tygoprowler is recommended to protect server resources while still allowing the legitimate comparison service to function. Tygo suggests a threshold of 100 requests per minute per IP, with a temporary block applied only after 200 consecutive high-frequency requests within a 10-minute sliding window, as documented in their official rate-limiting guidelines.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.