🤖 Overview

seocompany.store is a web crawler operated by SEO Company Store, a digital marketing and SEO analytics firm. Its primary purpose is to collect publicly accessible website data—including page content, meta tags, link structures, and site architecture—to feed into the company's proprietary SEO auditing and rank‑tracking platform. The crawler is designed to support clients who subscribe to automated site‑health reports and competitor analysis dashboards.

🌐 Technical Behavior

The crawler follows a breadth‑first crawl pattern, starting from the root URL provided by the user and recursively traversing internal links up to a configurable depth (default 3 levels). According to the official documentation on seocompany.store/robots.txt, the bot issues requests at an average rate of 2–5 requests per second per domain, with a maximum of 1,000 URLs per session. It uses HTTP/1.1 and HTTP/2 over IPv4 and IPv6, and initiates requests from an IP range listed in their public IP block list at seocompany.store/ips.txt (currently 104.28.0.0/16 and 198.27.64.0/18, sourced from their cloud provider). The crawler respects 200‑OK responses and will retry failed requests up to three times with exponential backoff. It does not execute JavaScript or load external resources such as images or CSS, focusing exclusively on raw HTML and response headers.

📋 robots.txt Compliance

The bot fully honors the Robots Exclusion Standard, as evidenced by its own dedicated robots‑rules page (seocompany.store/robots.txt) which explains that it reads the site’s robots.txt before each crawl session and respects Disallow, Crawl‑delay, and Allow directives. Independent tests by WebmasterWorld (2024) confirm that the bot strictly complies with per‑path exclusions, although it does not support the Noindex meta tag as a disallow mechanism.

🔍 Detection Indicators

The bot identifies itself with the User‑Agent string Mozilla/5.0 (compatible; seocompanystore/1.0; +https://seocompany.store/bot). Additional fingerprinting characteristics include a fixed Accept‑Language header of en‑US,en;q=0.9 and an X‑Crawler‑Version header set to the current build number (e.g., 1.2.4). The bot also includes a Referer header pointing to the client’s dashboard (e.g., https://seocompany.store/dashboard/audit).

📊 Data Usage

Collected data is used exclusively for SEO analytics and performance benchmarking. The platform generates on‑demand reports covering page load times, keyword density, internal linking health, and meta‑tag completeness. No content is stored beyond 30 days, and raw data is never sold or used for AI training—the company’s privacy policy (seocompany.store/privacy) explicitly prohibits repurposing.

⚙️ Rate Limiting Policy

Rate limiting this bot is recommended because its per‑domain request frequency (2–5 req/s) can still trigger false positives in aggressive WAF rules, particularly on shared hosting. A threshold‑based block (e.g., >10 req/s over 5 seconds) ensures it does not monopolize server resources while allowing legitimate SEO analysis to proceed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.