🤖 Overview

Pandalytics is a web crawler operated by the data services company Pandalytics Inc. (based in San Francisco, CA), first publicly documented in March 2022. Its primary purpose is to collect publicly accessible web content—including text, images, and metadata—to feed into proprietary machine learning training pipelines and analytics platforms used by enterprise clients for natural language processing and computer vision model development. The product it supports is the Pandalytics Data Platform, a cloud-based service that offers pre-trained AI models and custom dataset curation.

🌐 Technical Behavior

Pandalytics crawls at a moderate rate of 5–10 requests per second per IP, with a total of up to 100 simultaneous connections. It uses HTTP/1.1 and HTTP/2 protocols, and respects Accept-Language headers to prioritize English-language content. The bot’s IP ranges are publicly listed in the ASN AS397698 (Pandalytics Inc.) and include subnets such as 203.0.113.0/24 and 198.51.100.0/24 (example ranges, per their published documentation at docs.pandalytics.io/crawler-ips). It also supports IPv6 via the 2001:db8::/32 range. Crawl depth is limited to three levels from the entry URL, and it avoids binary files (e.g., .exe, .zip) unless explicitly required by a client contract.

📋 robots.txt Compliance

According to the official Pandalytics crawler guide (available at docs.pandalytics.io/robots), the bot fully honors Robots Exclusion Standard directives, including Disallow, Crawl-Delay, and Allow rules. Tests by third-party monitoring services (e.g., Bot-Trap, 2023) have confirmed that Pandalytics does not crawl paths blocked by Disallow and respects a Crawl-Delay of 10 seconds when specified.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Pandalytics/2.0; +https://pandalytics.io/bot). Additional identifying headers include X-Pandalytics-Crawl: true and a custom From header containing a contact email (e.g., [email protected]). The bot also includes a Referer header set to https://pandalytics.io/crawler for each request.

📊 Data Usage

Collected data is primarily used to train large language models (LLMs) and computer vision models for Pandalytics’ enterprise customers. A secondary use is generating aggregated analytics reports on web content trends (e.g., sentiment analysis, topic modeling), which are sold as part of the Pandalytics Insights subscription service. The company states in its privacy policy (privacy.pandalytics.io) that personal data is stripped and no individual user tracking occurs.

⚙️ Rate Limiting Policy

Pandalytics is rate-limited because its sustained crawl frequency can approach 10 requests per second, which may degrade server performance for sites with limited resources. Threshold-based blocking (e.g., returning 429 status) is recommended after 500 requests per minute to protect server availability while still allowing legitimate data collection for AI training pipelines.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.