🤖 Overview

ProPowerBot is a legitimate web crawler operated by ProPower Inc., a data analytics and AI training company headquartered in San Francisco, California. First documented in 2021, its primary purpose is to collect publicly available web content for training large language models and improving the company's proprietary natural language processing systems. The bot also aggregates product information, pricing data, and user reviews for commercial intelligence products sold by ProPower.

🌐 Technical Behavior

ProPowerBot performs HTTP requests using a custom Python-based crawler that respects standard HTTP protocol conventions. It issues requests with a configurable interval, typically between 2 to 10 seconds per domain, though it may burst up to 5 requests per second during initial discovery. The bot's IP ranges are sourced from Amazon Web Services (AWS) and Google Cloud Platform (GCP), with CIDR blocks 3.0.0.0/8 and 35.0.0.0/8 commonly observed. It identifies itself via the User-Agent string "ProPowerBot/1.0 (+https://propower.com/bot)" and also includes a "From: [email protected]" header. The crawler follows links recursively up to depth 10 and indexes HTML, PDF, and plain text content, while ignoring binary files. It does not execute JavaScript or submit forms.

📋 robots.txt Compliance

According to ProPower's official documentation at https://propower.com/robots-txt-policy, the bot fully honors Disallow directives found in robots.txt files. The documentation states that ProPowerBot will cache the robots.txt for up to 24 hours and re‑check before each crawl session. If a site returns a 403 or 503 status, the bot immediately ceases crawling that domain for at least 30 days. Compliance is verified through independent crawler audits.

🔍 Detection Indicators

The primary User‑Agent string is "ProPowerBot/1.0". The bot may also use "ProPowerBot/2.0" for internal testing. It always includes a "From" header and a "X-Robots-Tag: noindex" header on its own requests. The IP addresses originate from the ASNs of AWS (AS16509) and GCP (AS15169). Log analysis shows consistent request intervals and a lack of referrer header forgery.

📊 Data Usage

Collected data is used to train ProPower's proprietary LLMs, specifically the "PowerGPT" series, and to provide business intelligence dashboards for enterprise clients. The company publishes a transparency report at https://propower.com/transparency describing data sources and opt‑out mechanisms.

⚙️ Rate Limiting Policy

ProPowerBot is rate‑limited because its high crawl volume—up to 10 million pages per day—can degrade server performance for smaller websites. Threshold‑based blocking (e.g., more than 100 requests per minute) is recommended to prevent resource exhaustion while still allowing legitimate indexing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.