🤖 Overview

SearchPreview is a legitimate web crawler operated by SearchPreview Ltd. (searchpreview.com), designed to generate thumbnail previews of web pages for use when links are shared on social media, messaging apps, and search result snippets. First documented in 2015, its core function is to capture full-page screenshots using a headless browser, providing visual context without storing textual content for AI training or indexing. The bot operates under a public robots.txt policy and is widely recognized by webmasters.

🌐 Technical Behavior

The crawler uses a headless Chromium engine to render JavaScript and CSS, simulating a real browser. It issues HTTP GET requests at a rate of 10–15 requests per second per IP, sourced from a pool of over 200 IPs within ASN AS16509 (Amazon Web Services) and AS40676 (Psychz Networks) as listed in official documentation. Request timeout is 30 seconds, and the bot follows up to one redirect per URL, stopping at crawl depth 3. It respects gzip encoding and does not crawl binary files or deep nested directories.

📋 robots.txt Compliance

According to searchpreview.com/robots.txt, the bot fully honors Disallow directives and supports a configurable Crawl-Delay. It rechecks robots.txt every 24 hours per domain. Webmaster reports confirm immediate cessation on blocked paths, and the bot does not override user-agent exclusions.

🔍 Detection Indicators

Primary User-Agent: SearchPreview/2.0 (compatible; +http://searchpreview.com/bot). Also seen as SearchPreview/1.0. The bot sets an X-Bot header to "searchpreview". Behavioral fingerprint: consistent 2-second inter-request delay, preference for HTML over images or PDFs, and no concurrent requests to the same domain.

📊 Data Usage

Collected data is used solely to generate static preview images (PNG/JPEG) stored for up to 7 days. No textual content is retained, analyzed for AI training, or used for search rankings. The previews are served exclusively via the searchpreview.com CDN to enhance link sharing across platforms like Twitter, Slack, and Facebook.

⚙️ Rate Limiting Policy

Rate limiting is applied to prevent server overload; the bot recommends a crawl delay of 5 seconds for small sites and self‑throttles when encountering a 429 status code. It is safe to allow with threshold‑based blocking only if requests exceed 30 per minute to a single domain per its own operational guidelines.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.