🤖 Overview

Id-search is a web crawler operated by ID Search Ltd., a search engine company headquartered in Europe. It systematically indexes publicly accessible web pages to populate and maintain the ID Search engine's search index. The bot was first observed in early 2010 and has been updated periodically to improve crawl efficiency and adherence to web standards.

🌐 Technical Behavior

The crawler initiates HTTP GET requests using both HTTP/1.1 and HTTP/2 protocols. By default, it limits requests to one per five seconds per host, though it can increase this rate during targeted indexing bursts. The User‑Agent string is Mozilla/5.0 (compatible; Id‑search/1.0; +http://www.id‑search.org/bot.html). IP addresses are drawn from a distributed pool primarily allocated to EU‑based data centers, including ranges such as 185.200.0.0/16 and 217.73.0.0/16. Before crawling a site, it fetches the robots.txt file and caches the directives for up to 24 hours. Requests include typical HTTP headers like Accept: text/html and Accept‑Language: en,*, as well as a From: header containing the email address crawler@id‑search.org for site owner contact. The bot follows internal links first in a breadth‑first manner before moving to external links.

📋 robots.txt Compliance

According to the official documentation at www.id‑search.org/bot.html, Id‑search fully complies with the robots.txt standard. It honors both explicit Disallow directives and Crawl‑Delay settings. However, community reports on webmaster forums indicate occasional lapses when the robots.txt file has syntax errors or non‑standard patterns; the operator has stated they investigate and fix such issues promptly upon notification.

🔍 Detection Indicators

The primary detection signature is the User‑Agent string Id‑search/1.0 coupled with the email address in the From: header. Behavioral patterns include requesting robots.txt exactly once per crawl session, followed by a sequence of rapid page fetches from the same domain. Reverse DNS lookups on the crawler's IP addresses reveal hostnames in the format crawlerX.id‑search.net. Additionally, the bot is listed on user‑agent databases such as user‑agents.org with a brief description.

📊 Data Usage

Collected web page data is used exclusively to build and update the ID Search engine's index. The company's privacy policy states that page content is stored temporarily during the indexing process and is not retained for longer than necessary to refresh the index, which occurs approximately every two weeks for most pages. The data is not sold or used for training machine learning models or AI systems.

⚙️ Rate Limiting Policy

Although Id‑search is a legitimate search engine crawler, its burst traffic can still cause significant load on smaller web servers. A threshold‑based rate limiting policy—for example, allowing up to 120 requests per minute from a single IP before throttling—ensures the bot can still index the site while preventing resource exhaustion. This approach balances the need for comprehensive indexing with server stability.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.