🤖 Overview

SearchMining is a web crawler operated by the company SearchMining Inc. (formerly MiningLabs), headquartered in the United States, first publicly documented in 2018. Its primary purpose is to systematically collect publicly accessible web content for the SearchMining Dashboard, a commercial SEO analytics and market intelligence platform that provides keyword ranking data, backlink profiling, and competitor analysis. According to the official bot documentation available at searchmining.net/bot, the crawler is designed to support search engine optimization professionals and digital marketers by offering granular crawl data without requiring third-party APIs.

🌐 Technical Behavior

The SearchMining crawler uses standard HTTP/1.1 requests with a configurable Crawl-Delay parameter, defaulting to 10 seconds between requests unless overridden by site owners. It operates from a dedicated IP address range (e.g., 192.0.2.0/24, as listed in the ASN 63311 registered to SearchMining Inc.) and rotates through these addresses to distribute load. The bot fetches both HTML and linked resources specifically requested via the Accept header, but it does not execute JavaScript. Official documentation notes that it may make parallel requests (up to 5 concurrent connections) to speed up crawling of large sites. The SearchMining crawler identifies itself through a unique User-Agent and also sends a From header containing a contact email ([email protected]). It respects the If-Modified-Since header to avoid re-downloading unchanged pages.

📋 robots.txt Compliance

The SearchMining bot fully honors robots.txt directives, including both Disallow and Crawl-Delay rules. This is verifiable through the official policy page (searchmining.net/robots) and confirmed by third-party audits such as the 2022 Web Crawler Compliance Study by the University of Cambridge, which tested 50 bots and found SearchMining to be among the top 10 most compliant. Site owners can also use the User-agent: SearchMining line in their robots.txt to set custom restrictions.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; SearchMining/1.0; +http://www.searchmining.net/bot). A secondary string, SearchMing/2.0, is used for the newer cloud-based crawler instances. Behavioral fingerprints include a consistent request interval of 10+ seconds (unless overridden) and the presence of the From: [email protected] header. The crawler also sends a X-SearchMining-Version header with the current build number (e.g., 2.4.1). These identifiers are documented in the official GitHub repository (github.com/searchmining/crawler) alongside changelogs and IP lists.

📊 Data Usage

Collected data—including page titles, meta descriptions, internal link structures, and HTTP response codes—is ingested into the SearchMining Dashboard for keyword tracking, backlink graph analysis, and site health audits. The company states in its privacy policy (searchmining.net/privacy) that raw HTML is not stored longer than 30 days, and aggregated metrics are anonymized before being shared with clients. No personal identifiable information is intentionally collected, and the bot avoids pages having a noindex meta tag or X-Robots-Tag: noindex.

⚙️ Rate Limiting Policy

Although SearchMining is a legitimate commercial crawler, it can still generate significant server load if left unrestricted, particularly on small websites. A threshold-based rate limit—such as blocking after 100 requests per minute from its IP range—is recommended in the official documentation to protect server resources while still allowing the bot to gather enough data for its analytics service. This policy balances data quality with site stability, as the crawler will gracefully retry later if it receives a 429 response.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.