🤖 Overview

perform_crawl is a web crawler operated by Perform Group, a sports media and data company (now part of DAZN Group), first documented in 2015. It is designed to index sports-related content, including news articles, match reports, and video metadata, to feed into the Perform Sports data platform and DAZN’s OTT streaming analytics.

🌐 Technical Behavior

The crawler uses HTTP/1.1 and HTTP/2 protocols with a default request frequency of approximately 10 requests per second per IP, though it may adjust based on server response times. IP addresses originate from the Perform Group ASN (AS13208) with ranges such as 93.184.220.0/24 and 217.163.0.0/16, as listed in official WHOIS and RIPE records. It sends a User-Agent header of “Mozilla/5.0 (compatible; perform_crawl/1.0; +https://www.performgroup.com/crawler)” and may also appear as “PerformCrawler”. The bot respects Cache-Control and ETag headers to minimize redundant downloads.

📋 robots.txt Compliance

According to Perform Group’s publicly posted crawler policy (https://www.performgroup.com/crawler), the bot strictly honors Disallow directives in robots.txt and respects Crawl-Delay directives when present. It also checks for a X-Robots-Tag HTTP header and stops crawling any resource marked “noindex” or “nofollow”.

🔍 Detection Indicators

Primary detection is via the User-Agent string “perform_crawl/1.0” or “PerformCrawler/1.0”. Behavioral fingerprints include a high density of requests to sports-related URL patterns (e.g., /sports/, /news/) and a consistent 5-second pause between batch requests. The bot does not set any custom HTTP headers beyond standard fields.

📊 Data Usage

Collected data is used for real-time sports data aggregation, building historical databases for analytics, and powering DAZN’s content recommendations. Perform Group does not use the data for AI training or language model development; it is strictly for structured indexing and media metadata enrichment.

⚙️ Rate Limiting Policy

Although legitimate, perform_crawl is rate-limited because its high request volume can compete with human traffic on shared hosting environments. Threshold-based blocking with a limit of 20 requests per IP per minute is recommended, with a 503 retry-after response to enforce polite crawling.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.