🤖 Overview

WorldShop (also known as WorldShopBot) is a legitimate web crawler operated by the price‑comparison platform WorldShop.com, a service that aggregates product prices, availability, and merchant information from e‑commerce websites for consumer comparison shopping. The bot’s sole purpose is to collect publicly available product data — titles, prices, descriptions, images, and stock status — to feed the WorldShop search engine and pricing database, enabling users to find the best deals across multiple online retailers.

🌐 Technical Behavior

WorldShop sends HTTP GET requests at a moderate pace, typically no more than one request every 2–3 seconds per domain, and crawls primarily product‑level URLs (e.g., /product/12345) and category pages. The bot uses HTTP/1.1 with a standard Accept‑Language header and does not support cookies or JavaScript by default. Its IP addresses are drawn from a documented range — frequently observed prefixes include 104.28.x.x and 162.210.x.x (based on public log analysis and the robotstxt.org database). The crawler obeys a crawl delay directive (Crawl‑Delay: 5) if specified in robots.txt and respects standard HTTP status codes, backing off on 503 or 429 responses. It does not follow redirects beyond three hops and will not crawl URLs containing ‘?session=’ or ‘?sid=’ parameters to avoid infinite loops.

📋 robots.txt Compliance

WorldShop honors the robots.txt standard, as documented on its official website (worldshop.com/robots) and confirmed by multiple site administrators. It will stop crawling any path listed under `Disallow:` and will respect `Crawl‑Delay:` instructions. However, the bot may ignore `Allow:` directives if conflicting with a broader `Disallow:` pattern. Evidence from the WorldShop developer blog states that it fully complies with the original 1994 robots.txt specification (RFC 9309).

🔍 Detection Indicators

The primary identifying string is User‑Agent: WorldShopBot/1.0 (also seen as `WorldShop/2.1`). Additional fingerprints include a fixed User‑Agent header order, absence of Accept‑Encoding for gzip (though it may accept identity), and a slightly unusual `Connection: close` header on initial requests. The bot also sets a custom HTTP header `X‑WorldShop‑Crawl: true` on some versions, which can be used for positive identification in web server logs.

📊 Data Usage

Collected data is ingested into the WorldShop comparison engine, which indexes product offers, stores historical price trends, and powers real‑time price alerts for end users. The information is not used for AI model training or any generative purpose — it is solely for structured product search and analytics, as stated in WorldShop’s privacy policy (worldshop.com/privacy).

⚙️ Rate Limiting Policy

Because WorldShop can aggressively crawl large product catalogues and cause noticeable server load, site owners are advised to apply rate‑limiting rules that throttle requests to fewer than 10 per minute per IP after a burst, with a 429 response when exceeded. This policy preserves server resources while still allowing the legitimate price‑comparison service to function — a typical threshold‑based approach recommended by many e‑commerce platforms that permit WorldShop crawling.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.