🤖 Overview

AddSearchBot is a legitimate web crawler operated by AddSearch AB, a Swedish company providing site search and data extraction services since 2008. Its primary purpose is to index website content for AddSearch’s hosted search engine, enabling customers to offer fast, relevant site search on their own domains. The bot collects publicly accessible HTML pages, PDFs, and other text-based documents to build searchable indexes, and it is explicitly not used for AI training or advertising.

🌐 Technical Behavior

AddSearchBot crawls at a moderate, configurable rate — by default it issues one request every 1–2 seconds per domain, but customers can adjust this in the AddSearch dashboard. It respects the Link: rel="nofollow" attribute and X-Robots-Tag HTTP header. The bot uses HTTP/1.1 and HTTP/2, and identifies itself via the User-Agent header AddSearchBot/1.0 (or AddSearchBot/2.0 in newer versions). IP ranges are listed in AddSearch’s official documentation at https://addsearch.com/docs/crawler-ips and are also published in the _addsearch.txt file served from the customer’s domain. The bot follows HTTPS redirects and enforces a maximum crawl depth of 5 levels by default, though this is customisable.

📋 robots.txt Compliance

AddSearchBot fully complies with the Robots Exclusion Protocol (RFC 9309). Official documentation states that it reads and honours Disallow, Allow, Crawl-Delay, and Disallow: / directives. If a site blocks it via robots.txt, the crawler will stop immediately and not retry for at least 24 hours. There is no documented evidence of it ignoring or overriding robots.txt rules.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string: AddSearchBot/1.0 or AddSearchBot/2.0. Additionally, it sends a From header with the customer’s email address (if configured) and a Referer header pointing to the customer’s domain. The bot’s IP addresses are documented in AddSearch’s public IP list and belong to ranges allocated to AddSearch AB (ASN 200242). No other identifying headers are standard.

📊 Data Usage

Collected data is used exclusively to power site search for AddSearch’s customers. The indexed content remains private to the customer’s search index and is not shared with third parties, sold, or used for training large language models. Customers can delete their index at any time, and AddSearch does not retain raw content beyond the customer’s subscription period, per their privacy policy at https://addsearch.com/privacy.

⚙️ Rate Limiting Policy

Although AddSearchBot is legitimate and non‑malicious, it may still be rate‑limited by server administrators to prevent excessive load. A threshold‑based block (e.g., >10 requests per second) is a reasonable precaution because the bot’s default crawl rate is low, and any higher frequency likely indicates a misconfigured instance or a duplicate crawl request.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.