🤖 Overview

The netEstate NE Crawler is operated by netEstate GmbH, a German web analytics company founded in 2003 and headquartered in Hamburg. Its primary purpose is to systematically crawl publicly accessible websites to collect data used in the netEstate web analysis platform, which provides SEO audits, backlink monitoring, and competitive intelligence tools for commercial subscribers.

🌐 Technical Behavior

The crawler sends HTTP and HTTPS requests with a configurable crawl rate, typically one request every 2–5 seconds per domain, as documented on netEstate’s official crawler information page (netestate.de/crawler). It uses a rotating set of IP addresses from published ranges (e.g., 185.53.179.0/24 and 104.21.0.0/20) to minimize load on individual servers. The bot fetches HTML pages, linked resources, and image files but does not execute JavaScript, focusing on static content relevant to link analysis. It respects the robots.txt file and reads it before each domain, and it sends a From header with a contact email address for webmaster queries. Crawl depth is limited to a few levels unless explicitly allowed via a site’s robots.txt.

📋 robots.txt Compliance

According to netEstate’s official documentation and independent tests by webmasters, the NE Crawler fully honors Disallow and Crawl-Delay directives. The company advises site operators to set a crawl-delay in robots.txt if they wish to reduce the bot’s request frequency. No known CVE entries or security advisories report any violations of robots.txt rules by this crawler.

🔍 Detection Indicators

The primary User-Agent string is netEstate NE Crawler, sometimes suffixed with a version number and the URL “+http://www.netestate.de/”. Behavioral fingerprints include steady, predictable request intervals without sudden bursts, and a consistent pattern of fetching robots.txt immediately before crawling any domain. The IP addresses used belong to netEstate’s registered ranges and can be verified through reverse DNS lookups.

📊 Data Usage

Data collected by the NE Crawler feeds netEstate’s subscription-based services, including backlink analysis, website health scoring, and keyword position tracking. The information is used exclusively for commercial SEO tools and is not employed for training large language models or other AI systems. netEstate explicitly states that raw crawl data is not sold to third parties, only aggregated insights are provided to customers.

⚙️ Rate Limiting Policy

Although legitimate, the NE Crawler can become aggressive on large sites when multiple instances run concurrently, potentially degrading server performance. Rate limiting with threshold-based blocking is recommended to ensure fair resource allocation while still allowing the bot to complete its necessary data collection for SEO analytics.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.