🤖 Overview

SEOkicks-Robot is a legitimate web crawler operated by SEOkicks, a German-based SEO software company founded in 2009. It is used to collect publicly accessible website data for SEO analysis, including backlink profiles, page metrics, and keyword rankings, feeding into the SEOkicks platform. The bot is designed to help webmasters and SEO professionals monitor their own sites and competitors, as documented in SEOkicks’ official documentation and robotstxt.org.

🌐 Technical Behavior

The crawler operates over HTTP/1.1 and HTTPS protocols, with requests originating from IP ranges primarily registered to Hetzner Online GmbH in Germany (e.g., 5.9.0.0/16 and 78.46.0.0/16). It typically sends one request every 5 to 10 seconds per host, with a configurable crawl delay that respects Crawl-Delay directives. SEOkicks-Robot fetches HTML pages, CSS, JavaScript, and image files to analyze page structure and link attributes, but does not execute JavaScript for rendering. It follows standard HTTP status codes and will back off on 429 responses. The bot uses a modified version of the Apache HTTP Client and does not support cookies or session persistence.

📋 robots.txt Compliance

According to SEOkicks’ published documentation and verified by multiple webmaster forums, SEOkicks-Robot fully honors robots.txt directives, including Disallow, Allow, and Crawl-Delay. It re-fetches the robots.txt file at least once every 24 hours or on cache expiry. The bot is listed in the official robots.txt database with the user-agent name SEOkicks-Robot, and operators can block it globally by adding User-agent: SEOkicks-Robot Disallow: /.

🔍 Detection Indicators

The primary identifying User-Agent string is: Mozilla/5.0 (compatible; SEOkicks-Robot; +https://www.seokicks.de/robots.html). Variants may include a version number or platform identifier. Behavioral fingerprints include requesting /robots.txt first, a consistent request pattern without random delays, and absence of a Referer header. The bot also sends a Via header sometimes indicating "SEOkicks Crawler".

📊 Data Usage

Collected data is used for SEO analysis and reporting within the SEOkicks SaaS platform. This includes backlink monitoring, on-page SEO audits, keyword position tracking, and competitor comparison. Data is aggregated and anonymized before presentation; individual user sessions or PII are not stored. SEOkicks explicitly states they do not sell data to third parties; data is used solely for internal tooling and client dashboards.

⚙️ Rate Limiting Policy

Although SEOkicks-Robot is legitimate and rate-limited by design, aggressive crawl patterns on smaller sites may warrant threshold-based blocking. Webmasters can set a crawl delay or block the bot entirely if its requests impact server performance, but blocking should be a last resort after confirming the bot is not malicious. The policy rationale is to prioritize site stability while allowing beneficial SEO data collection.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.