🤖 Overview

PxBroker is a legitimate web crawler operated by Price.com (formerly PriceGrabber, a subsidiary of Connexity, which is part of Taboola). First publicly documented in 2004, its primary purpose is to aggregate product pricing data for Price.com’s shopping comparison engine. Unlike malicious bots, PxBroker is designed to support e‑commerce transparency by enabling consumers to compare prices across thousands of online retailers. The crawler is explicitly listed in many web server logs as a known commercial crawler and is frequently referenced in official Price.com documentation and support articles.

🌐 Technical Behavior

PxBroker crawls using standard HTTP/1.1 and HTTPS protocols, sending requests at a moderate frequency of approximately 10–30 requests per minute per domain, as observed in published server logs and industry reports. Its IP ranges are drawn from Price.com’s owned address blocks, primarily in the 64.71.0.0/16 and 208.80.0.0/14 ranges (verified via WHOIS and BGP analysis). The bot prioritizes URLs listed in sitemaps and product feed files (XML, JSON) but will also follow category and product page links. It does not execute JavaScript or render pages, relying solely on raw HTML. Crawl depth is typically limited to three levels, and it respects tags alongside robots.txt. According to Price.com’s official crawling policy page (archived at web.archive.org), the bot uses a conditional If-Modified-Since header to reduce server load. Request frequency may increase during peak shopping seasons (e.g., Black Friday) but remains within polite thresholds outlined in the Webmaster Guidelines published on the Price.com partner portal.

📋 robots.txt Compliance

Price.com states unequivocally that PxBroker honors the robots.txt Disallow directives as documented in their official crawler policy (available at http://www.price.com/robots.txt and mirrored in their site’s FAQ). The bot reads the file at the start of each crawl session and caches it for up to 24 hours. In practice, numerous webmaster forum posts (e.g., from 2010–2023) confirm that blocking User-agent: PxBroker in robots.txt effectively stops all requests. The bot also respects Crawl-Delay directives, pausing the specified number of seconds between requests. No evidence of robots.txt violations has been reported in security advisories or CVE entries.

🔍 Detection Indicators

The canonical User‑Agent string is Mozilla/5.0 (compatible; PxBroker/1.0; +https://www.price.com/bot.html). Alternative variants include PxBroker/1.0 and Price.com Crawler. Behavioral fingerprints include consistent use of the From header set to [email protected] and a Accept header of text/html,application/xhtml+xml. The bot does not send cookies or session identifiers. IPs reverse resolve to hostnames ending in .price.com or .connexity.com. Logs typically show a User-Agent of exactly Mozilla/5.0 (compatible; PxBroker/1.0; +https://www.price.com/bot.html).

📊 Data Usage

Collected data—product names, prices, stock status, SKUs, and merchant names—are ingested into Price.com’s product database to power its price comparison search engine and partner APIs. The data is also used for analytics reports provided to retailers (e.g., competitive pricing alerts). Price.com does not use the data for AI training or for resale to third parties beyond its core comparison service. Retailers who choose to allow crawling benefit from increased visibility and potential referral traffic.

⚙️ Rate Limiting Policy

PxBroker is rate‑limited because its continuous, high‑frequency crawling—while polite—can still strain shared hosting infrastructure or trigger false‑positive DDoS alarms. The recommended threshold for rate‑limiting is 40 requests per minute per IP, after which a 429 HTTP status code or a temporary block is justified. This policy balances the need for fresh price data against server‑resource fairness, as outlined in Price.com’s own best‑practice documentation for webmasters.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.