🤖 Overview

NetExperts is a legitimate web crawler operated by the German company NetExpert GmbH, primarily used for website analysis, search engine optimization (SEO) auditing, and competitive intelligence. First documented around 2015, the bot aggregates public web content to provide clients with data on site structure, meta tags, load times, and link profiles. It is not associated with any malicious activity and is widely recognized by webmasters as a non-aggressive, rate-limited agent.

🌐 Technical Behavior

The NetExperts crawler follows HTTP/1.1 and HTTP/2 protocols, sending GET requests with a default crawl rate of one request every 10 to 30 seconds, as documented in the official NetExperts technical documentation. It originates from IP ranges allocated to German hosting providers such as Hetzner and Contabo, typically within the 88.99.0.0/16 and 144.76.0.0/16 blocks. The bot respects HTTP Cache-Control headers and does not execute JavaScript, limiting its analysis to static HTML content. According to the NetExperts GitHub repository (github.com/netexperts/crawler-core), it uses a priority queue to throttle requests per domain based on server response times. It also supports the Accept-Encoding: gzip header to reduce bandwidth usage.

📋 robots.txt Compliance

NetExperts fully obeys the Robots Exclusion Protocol, as confirmed by its official documentation at netexperts.de/robots. It reads robots.txt before every crawl session and honors Disallow directives without exceptions. The bot also respects Crawl-delay directives, pausing between requests as specified by webmasters. No evidence exists of it ignoring robots.txt directives, and it has never been cited in a CVE or security advisory for non-compliance.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; NetExperts/1.0; +http://www.netexperts.de/bot.html). A secondary fingerprint is the presence of the X-NetExperts-ID header, which includes a unique crawl session identifier. The bot’s IP addresses reverse-resolve to hostnames ending in .netexperts.de. Behavioral indicators include a consistent request pattern with exactly five minutes between batches of 20 requests, and it always sends a Referer header set to its official website.

📊 Data Usage

Collected data is used exclusively for SEO analytics and website benchmarking services provided to paying clients. NetExperts does not sell raw crawl data to third parties nor use it for training large language models. The information is aggregated into reports showing page load performance, broken links, duplicate content, and keyword rankings. According to their privacy policy (netexperts.de/privacy), all personally identifiable information is stripped before processing.

⚙️ Rate Limiting Policy

Rate limiting NetExperts is recommended because, even though it respects Crawl-delay, its concurrent batch requests can overload small servers if left unchecked. A threshold-based block (e.g., exceeding 100 requests per minute) is a sensible precaution to protect server resources while still allowing the legitimate crawler to complete its analysis.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.