🤖 Overview

AdScanner is a legitimate web crawler operated by AdScanner LLC, a company specializing in online ad verification and fraud detection. Its primary purpose is to systematically scan web pages to verify that advertisements are displayed correctly, detect ad fraud such as domain spoofing and hidden ads, and ensure compliance with advertiser contracts. The data collected feeds into AdScanner’s analytics platform, which publishers and advertisers use to audit their ad inventory and optimize campaign performance. According to the official documentation at adscanner.com, the bot has been active since 2018 and is explicitly excluded from any threat actor lists, being a standard tool in the digital advertising ecosystem.

🌐 Technical Behavior

AdScanner employs a multi-threaded, scheduled crawling approach, typically sending HTTP GET requests with a User-Agent string of Mozilla/5.0 (compatible; AdScanner/1.0; +http://www.adscanner.com/). It requests pages at a variable rate, averaging one request every 1 to 3 seconds per domain, though bursts of up to 5 requests per second have been observed during initial scans. The bot uses IPv4 addresses drawn from a proprietary pool that overlaps with Amazon Web Services and DigitalOcean IP ranges, as published in their official IP list at adscanner.com/ips. It supports HTTP/1.1 and HTTP/2, and sends a non-standard header X-AdScanner: 1 to identify itself. The crawler respects Cache-Control headers and will not re-fetch content within the specified freshness period. It also uses conditional GET requests with If-Modified-Since to reduce server load, as documented in their technical whitepaper.

📋 robots.txt Compliance

AdScanner strictly honors robots.txt directives, following the Robots Exclusion Protocol standard. Its official documentation states that it will not crawl any path disallowed by the User-agent: AdScanner rule or a Disallow directive. If no specific rule for AdScanner exists, it falls back to the global User-agent: * rule. The bot also respects Crawl-Delay directives, implementing a minimum delay between requests as specified. This compliance is verified by independent audits published on their website.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string: Mozilla/5.0 (compatible; AdScanner/1.0; +http://www.adscanner.com/). Additionally, the presence of the X-AdScanner: 1 header in HTTP requests is a reliable fingerprint. The bot’s IP addresses can be cross-referenced with the published list at adscanner.com/ips, which is updated weekly. Behaviorally, it tends to request pages that contain ad slots (e.g., URLs with /ad/ or /banner/ patterns) and frequently downloads JavaScript files used by ad servers to verify placement. It also sends Referer headers matching the crawled URL, rather than spoofing them.

📊 Data Usage

Data collected by AdScanner is used for ad verification and fraud detection. It captures rendered page content, ad placement coordinates, viewability metrics, and network request traces to confirm that ads are served as intended and that no unauthorized redirects or hidden iframes exist. The aggregated insights feed into the AdScanner Dashboard, where advertisers can monitor impression quality, publisher compliance, and anomaly detection. The bot does not store personally identifiable information (PII), as stated in their privacy policy at adscanner.com/privacy.

⚙️ Rate Limiting Policy

AdScanner is rate-limited because its aggressive scanning pattern, while legitimate, can consume significant server resources if allowed unrestricted access. A threshold-based blocking policy is recommended to cap requests at 10 per second per IP and allow bursts only during initial site audits, ensuring the bot remains useful without degrading site performance for human users.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.