🤖 Overview

Pattern is a legitimate web crawler operated by Pattern, Inc. (pattern.com), a company specializing in e‑commerce market intelligence and product data analytics. The bot systematically collects product listings, pricing, availability, and description information from publicly accessible retail websites to feed into Pattern’s proprietary analytics platform, which is used by brands and retailers for competitive benchmarking and demand forecasting. According to Pattern’s official documentation, the crawler is designed exclusively for collecting product‑level e‑commerce data and does not index personal or private content.

🌐 Technical Behavior

Pattern’s crawler operates using a distributed architecture that fetches multiple pages concurrently from the same domain, but it enforces a configurable per‑domain crawl delay to minimise impact on origin servers. The bot typically requests pages over HTTP/1.1 and HTTP/2, and it supports both compressed (gzip) and uncompressed content. Official documentation states that the crawler may issue up to several thousand requests per day on high‑volume retail sites, but it respects the Crawl‑Delay directive in robots.txt. IP address ranges are not publicly documented, but user reports and security logs indicate the crawler originates from a dynamic pool of IPs allocated to Pattern’s cloud infrastructure, primarily hosted on AWS and Google Cloud Platform. The bot does not follow client‑side JavaScript redirects or submit forms; it strictly parses static HTML and structured data (e.g., JSON‑LD, Microdata).

📋 robots.txt Compliance

Pattern explicitly states in its official user‑agent policy (accessible at pattern.com/robots) that the crawler fully honours Disallow directives found in a site’s robots.txt file. It checks the file at the start of each crawl session and will not revisit the robots.txt during the same session unless explicitly instructed. Evidence from multiple webmaster forums confirms that Pattern adheres to Crawl‑Delay values specified in robots.txt, reducing request frequency accordingly.

🔍 Detection Indicators

The primary User‑Agent string used by Pattern is "Mozilla/5.0 (compatible; PatternBot/1.0; +https://pattern.com/patternbot)". Alternative strings include "Pattern/1.0" and "PatternBot/2.0". The bot sometimes sets the From header to a contact email (e.g., [email protected]) and always includes a User‑Agent that explicitly identifies itself. Behaviourally, it exclusively requests GET methods, never sends cookies or authentication tokens, and ignores Set‑Cookie responses.

📊 Data Usage

Collected data is used exclusively for Pattern’s e‑commerce intelligence products, including real‑time pricing alerts, historical trend analysis, and inventory forecasting. The company explicitly states that personal information (names, emails, accounts) is neither collected nor stored. Pattern also aggregates anonymised data to produce market‑level reports for its clients. No data is used for AI training or general search indexing; the crawler’s sole purpose is product catalog analysis.

⚙️ Rate Limiting Policy

Pattern is rate‑limited on many commercial sites because its high‑frequency requests, while legitimate, can still degrade performance for other users. Threshold‑based blocking (e.g., returning 429 or 503 after a burst of requests) is an accepted standard to protect server resources without permanently banning the crawler, which respects those limits by backing off and retrying later.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.