🤖 Overview

cookienet is a legitimate web crawler operated by CookieNet Ltd., a UK-based privacy technology company. Its primary purpose is to scan websites for cookie declarations, third‑party trackers, and consent management platform (CMP) configurations, feeding data into CookieNet’s compliance dashboard used by organisations to monitor GDPR, ePrivacy, and CCPA adherence. The bot was first publicly documented in 2020 and is explicitly listed in official documentation at cookienet.com/crawler.

🌐 Technical Behavior

The crawler initiates HTTP/1.1 GET requests from a dynamic pool of IPv4 addresses belonging to AWS EC2 (us‑east‑1 and eu‑west‑1 regions) and DigitalOcean datacenters, with reverse DNS records typically resolving to crawler.cookienet.com. Requests are sent at a controlled rate of approximately one per 1.5 seconds per IP, with bursts not exceeding 15 requests per minute observed in production logs. The crawler respects Cache‑Control headers and avoids crawling pages that respond with 429 Too Many Requests. It uses TLS 1.2+ and sends a From header containing [email protected] for contact purposes.

📋 robots.txt Compliance

Cookienet fully honours Disallow directives in robots.txt, as confirmed by both its official policy at cookienet.com/bot-policy and independent testing by webmasters. It additionally respects Crawl‑Delay directives, pausing for the specified number of seconds. No evidence of ignoring robots.txt has been reported in security forums or CVE databases.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; cookienet/1.0; +https://cookienet.com/bot), with a secondary variant cookienet/1.0 (+https://cookienet.com/bot) used for non‑browser HTTP libraries. Behavioural fingerprints include a unique X‑Crawler‑ID: cookienet header and a consistent request pattern that always fetches /robots.txt before any other page on a given host.

📊 Data Usage

The collected data—cookie names, expiry dates, domain attributes, and CMP scripts—is aggregated into CookieNet’s cloud platform to generate automated compliance reports. These reports help website owners identify missing or incorrect cookie banners, cross‑domain tracking issues, and consent‑bypass attempts. Raw crawl data is retained for 90 days and is not used for AI training or advertising profiling.

⚙️ Rate Limiting Policy

Cookienet is rate‑limited because its automated scanning can generate a noticeable load on smaller sites if left unchecked; a threshold‑based block after 30 requests in 60 seconds is recommended to prevent resource exhaustion while still allowing the crawler to complete its legitimate compliance assessment.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.