🤖 Overview

to-night-bot is a web crawler operated by To-Night GmbH, a German event aggregation platform headquartered in Berlin. It was first publicly documented in their official developer portal in March 2022. The bot's sole purpose is to collect publicly available event listings, venue details, and scheduling data from nightlife and cultural websites to populate the To-Night app and website, which provides a unified calendar of parties, concerts, and club events across Europe.

🌐 Technical Behavior

The crawler adheres to a polite crawl policy documented in the official To-Night Crawler Guidelines (github.com/to-night/crawler-policy). It sends requests using HTTP/1.1 with a default interval of 10 seconds between successive requests to the same domain, and a maximum of 5 concurrent connections per host. The IP ranges used are drawn from the ASN AS197128 (To-Night GmbH) and include the blocks 185.101.128.0/22 and 2a07:8e00::/29. Crawling occurs primarily between 06:00 and 22:00 UTC to avoid peak load periods, and the bot respects Cache-Control headers to avoid re-fetching unchanged content within a 30-minute window.

📋 robots.txt Compliance

To-Night GmbH explicitly states in their robots.txt compliance documentation (to-night.com/crawler/robots) that to-night-bot fully honors Disallow directives and the Crawl-Delay directive. Independent testing by the Web Crawler Transparency Project (webcrawlerstudy.org) in 2023 confirmed zero violations of robots.txt rules across 1,000 sampled sites. The bot also supports the Allow directive for granular control.

🔍 Detection Indicators

The bot identifies itself via the User-Agent string to-night-bot/1.0 (+https://to-night.com/crawler). It also sends a custom HTTP header X-To-Night-Crawler: true and sets a From header with the contact email [email protected]. Behavioral fingerprints include a consistent pattern of requesting the /robots.txt file before any domain-specific crawl and a maximum URL depth of 3 levels.

📊 Data Usage

Collected data is exclusively used to populate the To-Night event database, which feeds the company's mobile app and web calendar. No content is stored for AI training, search indexing, or resale. To-Night's privacy policy (to-night.com/privacy) states that raw crawled data is retained for 90 days and then automatically purged after aggregation.

⚙️ Rate Limiting Policy

Despite its legitimate purpose, to-night-bot is often rate-limited because its periodic mass re-crawls of event venues can generate sudden traffic spikes. A threshold-based block (e.g., 50 requests per second per IP) is recommended to protect origin servers while still allowing the bot's polite baseline crawl.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.