🤖 Overview

Brandwatch is a web crawler operated by Brandwatch, a subsidiary of Cision, and is the primary data collection agent for the Brandwatch Consumer Intelligence platform. Its purpose is to systematically harvest publicly available content from websites, forums, blogs, news outlets, and social media platforms to feed Brandwatch’s analytics engine, which provides brand monitoring, sentiment analysis, and market research insights. The crawler is a legitimate, automated agent designed to support enterprise-level social listening and digital marketing intelligence, not a malicious actor.

🌐 Technical Behavior

The Brandwatch crawler performs broad scans of public web content, typically following links from seed URLs or sitemaps to discover relevant pages. It requests pages sequentially using a configurable crawl rate that can be slowed to avoid overloading servers; the default crawl frequency is approximately one request per second, but it can adjust based on the site’s response and robots.txt directives. The crawler operates primarily over HTTP/1.1 and HTTPS and uses a pool of IP addresses that are assigned from Brandwatch’s own address space—often from ASN 136168 (Brandwatch Ltd) and ASN 16509 (Amazon Web Services) when running on cloud infrastructure. While Brandwatch does not publicly list exact IP ranges, network operators can identify requests by the user-agent and reverse DNS lookups (e.g., crawler.brandwatch.com). The crawler respects standard HTTP caching headers (Last-Modified, ETag) and does not submit forms or engage in authenticated sessions.

📋 robots.txt Compliance

Brandwatch states in its official public documentation that the crawler fully respects robots.txt directives, including both User-agent and Disallow rules. Site owners can block the crawler by adding a line such as User-agent: Brandwatch followed by Disallow: /. Third-party tests and community reports confirm that Brandwatch adheres to these rules, though it may take a few hours for changes to propagate through its crawl queue.

🔍 Detection Indicators

The primary identifying User-Agent string is Brandwatch (e.g., Mozilla/5.0 (compatible; Brandwatch/2.0; +https://www.brandwatch.com/legal/crawler/)). In addition, the crawler may include an X-Robot-Identity header set to Brandwatch and a From header containing a contact email (e.g., [email protected]). Behavioral fingerprints include consecutive requests to a single domain with identical user-agent and a consistent crawl interval of roughly 1–2 seconds.

📊 Data Usage

All content collected by the Brandwatch crawler is ingested into the Brandwatch Consumer Intelligence platform, where it is processed and analyzed to produce real-time dashboards, trend reports, and alerts for brand managers and marketers. The data powers sentiment analysis, keyword tracking, competitor benchmarking, and influencer identification—never for general-purpose AI training or resale. Brandwatch’s privacy policy states that only publicly available data is collected and that personal information is anonymized before storage.

⚙️ Rate Limiting Policy

Because the Brandwatch crawler can generate a high volume of requests when scanning large sites—especially those with many pages—it is rate-limited to protect server performance and bandwidth. A reasonable threshold for blocking is 5–10 requests per second per IP, with a temporary ban applied if the crawler fails to honor 429 Too Many Requests responses. This policy ensures fair resource usage without cutting off a legitimate data source for brand monitoring.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.