🤖 Overview

DSearch is a web crawler operated by DuckDuckGo, the privacy-focused search engine, used to index web pages for its search results. It is also known by its primary user-agent string DuckDuckBot/1.0 and was first introduced in 2008 as part of DuckDuckGo's commitment to unbiased search indexing. The bot collects publicly accessible content to build and update the search engine's index, ensuring up-to-date results for users.

🌐 Technical Behavior

DSearch employs a breadth-first crawl strategy, respecting standard crawl delays and following links recursively from seed URLs. It sends requests over HTTP/1.1 and HTTP/2 protocols, with an average request frequency of 10-20 requests per second per IP, though this can vary based on site responsiveness. The bot originates from IP addresses belonging to Amazon Web Services (AWS), with ranges such as 54.196.0.0/16 and 54.198.0.0/16, as documented on DuckDuckGo's official crawler information page. It honors If-Modified-Since headers to reduce bandwidth usage and identifies itself via a User-Agent string that includes contact information for site operators.

📋 robots.txt Compliance

According to DuckDuckGo's official documentation at https://duckduckgo.com/duckduckbot, DSearch fully complies with the Robots Exclusion Protocol (REP). It reads robots.txt directives before crawling and respects Disallow rules, including those targeting specific paths or entire sites. Evidence from webmaster forums confirms that DuckDuckGo’s crawler consistently obeys these directives, and site owners can block it entirely using User-agent: DuckDuckBot followed by Disallow: /.

🔍 Detection Indicators

DSearch can be identified by its primary User-Agent string: Mozilla/5.0 (compatible; DuckDuckBot/1.0; +https://duckduckgo.com/duckduckbot). A secondary User-Agent for favicon crawling is DuckDuckGo-Favicons-Bot/1.0. It may also include a From header with the operator’s email ([email protected]) and a Referer header pointing to the crawling policy URL. Behavioral fingerprints include consistent request intervals and a pattern of accessing robots.txt before any other resource.

📊 Data Usage

The data collected by DSearch is used exclusively to populate and refresh DuckDuckGo's search index, which serves anonymous search results to users. DuckDuckGo does not use this data for AI training or personalized advertising, in line with its privacy policy. The index is updated continuously, with fresh content prioritized based on site update frequency and popularity signals.

⚙️ Rate Limiting Policy

DSearch is rate-limited because its crawl can be aggressive, sometimes exceeding 20 requests per second on large sites, which may impact server performance. Administrators are advised to set threshold-based blocking if the bot’s traffic disrupts normal operations, though it is not considered malicious and responds well to standard rate-limiting mechanisms like 429 Too Many Requests responses.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.