🤖 Overview

WebMeup-Crawler is a web crawler operated by the SEO and link analysis company WebMeup, primarily used to collect backlink data and website metrics for the WebMeup Backlink Checker tool, as documented on their official website at webmeup.com. The crawler systematically indexes links across the web to provide link-building insights and domain authority scores to paying subscribers, functioning as a legitimate SEO intelligence agent rather than a malicious scraper.

🌐 Technical Behavior

The WebMeup-Crawler performs regular full-site scans, often revisiting previously crawled pages every few weeks to detect new or changed backlinks, with request rates varying by site but generally respecting a delay of several seconds between requests to avoid overwhelming servers, according to WebMeup’s own crawl policy page. It makes HTTP/1.1 GET requests supporting both IPv4 and IPv6, and its IP ranges are associated with datacenters in the United States and Europe, though specific CIDR blocks are not publicly documented. The crawler typically follows no JavaScript execution, focusing solely on static HTML and meta tags for link extraction, and does not submit forms or interact with dynamic content.

📋 robots.txt Compliance

According to WebMeup’s official documentation and independent tests, the WebMeup-Crawler fully honors robots.txt Disallow directives, including those with wildcard patterns, and respects the Crawl-delay directive if set. The company explicitly states that if a site blocks the crawler via robots.txt, it will not revisit until the block is removed, as verified by their support article “How to Block WebMeup Crawler.”

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; WebMeup-Crawler/1.0; +https://webmeup.com/crawler.html), with a backup UA of webmeup-crawler used in some legacy scans, as noted in their official documentation. Additional identifying headers include a From field pointing to a support email and a Referer sometimes set to webmeup.com, but no custom X-Robots-Tag is sent.

📊 Data Usage

Collected data—such as outbound and inbound link URLs, anchor text, and page titles—is exclusively used to populate the WebMeup Backlink Checker database, providing subscribers with SEO reports and competitor backlink analysis. No collected content is used for AI training, general search indexing, or resale to third parties, as stated in their privacy policy.

⚙️ Rate Limiting Policy

WebMeup recommends rate-limiting the crawler to prevent excessive load, as its scans can be aggressive on high-link-density pages, and threshold-based blocking (e.g., 500 requests per hour per IP) is a reasonable policy to maintain site performance without completely denying access to a legitimate SEO tool.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.