🤖 Overview

SearchMarking is a legitimate web crawler operated by SearchMarking Inc., a digital marketing analytics company specializing in SEO intelligence and competitive research. First documented in 2019, this bot systematically indexes publicly accessible web pages to collect data on backlinks, keyword usage, content structure, and site performance metrics, which are then aggregated into the SearchMarking platform for actionable marketing reports. It is not associated with any threat actors and is designed to support ethical data collection for business analytics.

🌐 Technical Behavior

The SearchMarking bot primarily follows a scheduled crawl pattern, visiting websites at intervals ranging from a few hours to several days depending on the site’s update frequency and authority. Requests are made over TCP port 80 and 443 using HTTP/1.1, often with the Accept-Encoding: gzip header to reduce bandwidth consumption. The bot rotates through a pool of IP addresses predominantly hosted on Amazon Web Services (AWS) and DigitalOcean, as confirmed by reverse DNS lookups and public IP reputation lists. Crawl depth is typically limited to five levels from the entry point, and the bot respects robots.txt directives by default, though it may occasionally follow sitemap.xml links when present. Request frequency is moderate, averaging 2–5 requests per second per target domain, but can spike during initial discovery phases. The bot does not execute JavaScript or render interactive elements, relying solely on raw HTML responses.

📋 robots.txt Compliance

According to SearchMarking’s official documentation (available at searchmarking.com/robots-policy), the bot fully supports the Robots Exclusion Standard and will honor all Disallow directives found in the root robots.txt file. The bot also respects Crawl-Delay headers if explicitly set, pausing between requests accordingly. Empirical tests from website administrators confirm that blocking the bot via robots.txt reliably stops all subsequent crawling activity within the same crawl session.

🔍 Detection Indicators

The primary User-Agent string is "SearchMarking/1.0 (compatible; +http://searchmarking.com/bot)" and variants such as "SearchMarking/2.0 (compatible; +https://searchmarking.com)" for newer crawler versions. Additional identifying fingerprints include a custom HTTP header X-SearchMarking-ID containing a unique numeric token for each crawl session, and a consistent Referer header pointing to http://searchmarking.com. The bot’s reverse DNS hostname typically resolves to a pattern like searchmarking-*.aws-region.compute.amazonaws.com.

📊 Data Usage

Collected data—including page titles, meta descriptions, heading tags, backlink anchors, and internal link structures—is used exclusively for SEO competitive analysis, keyword opportunity identification, and site health scoring within the SearchMarking dashboard. No personal or sensitive user data is stored; the bot ignores login pages, forms, and cookie consent dialogs. The aggregated results are sold as subscription-based reports to marketing professionals and agencies seeking to benchmark their web presence against competitors.

⚙️ Rate Limiting Policy

Although legitimate, SearchMarking is rate-limited because its systematic, distributed crawling can inadvertently saturate server resources on smaller websites or those with limited bandwidth. Threshold-based blocking (e.g., limiting to 10 requests per minute from any single IP range) is recommended to maintain fair resource allocation without permanently denying the bot access, which would break its benign data collection function.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.