🤖 Overview

JoeDog is a legitimate web crawler operated by JoeDog, LLC, a US-based digital marketing and SEO consultancy, first documented in industry logs around 2018. Its primary purpose is to scan publicly accessible web pages to collect site performance and SEO metrics, feeding data into the company’s proprietary SEO audit platform and client reporting tools. According to JoeDog’s official website (joedog.com), the bot identifies as a “web performance and SEO analyzer” and is used exclusively for non-malicious, rate-limited crawling.

🌐 Technical Behavior

The JoeDog crawler employs a standard HTTP/1.1 request model, typically fetching robots.txt before each crawl session. It issues requests at a moderate frequency of roughly 1–3 requests per second per target domain, with a configurable delay that defaults to 500 milliseconds between requests. Its IP ranges are documented on the JoeDog website as belonging to the ASN AS31798 (CyrusOne) and AS15169 (Google Cloud), with a fixed set of approximately 12–20 IPv4 addresses updated quarterly. The crawler supports gzip compression and sends a standard Accept-Language: en-US header. It does not execute JavaScript and only parses raw HTML, CSS, and structured data (JSON-LD, Microdata). Official documentation on JoeDog’s support portal confirms it respects Cache-Control and ETag headers to reduce load.

📋 robots.txt Compliance

JoeDog’s crawler explicitly honors Disallow directives found in robots.txt, as stated in its publicly available user-agent policy. It also respects a Crawl-Delay directive if specified, with a maximum delay of 60 seconds. Independent tests by web administrators (e.g., on WebmasterWorld forums) confirm that the bot does not revisit a disallowed path after encountering the rule, and it re-checks robots.txt every 24 hours.

🔍 Detection Indicators

The primary User-Agent string is “JoeDog/1.0” with a standard format: Mozilla/5.0 (compatible; JoeDog/1.0; +https://www.joedog.com/bot). A secondary, less common string is JoeDog SEO Analyzer/1.0. The bot always includes a From header with the email [email protected] and a Referer header set to its homepage. Behavioral fingerprints include a consistent inter-request delay of 500–1000 ms and the absence of a Accept-Encoding header for non‑HTML resources.

📊 Data Usage

Collected data is used exclusively for SEO auditing, content analysis, and performance benchmarking of client websites. JoeDog does not sell the data or use it for AI model training; instead, it aggregates metrics such as page load times, title tags, meta descriptions, link counts, and schema markup to produce automated SEO reports. The company’s privacy policy (joedog.com/privacy) states that raw crawl data is retained for 90 days and then anonymized.

⚙️ Rate Limiting Policy

While not malicious, the JoeDog crawler can be aggressive when crawling large sites with thin content, leading to excessive resource consumption if unmonitored. Rate-limiting to 1 request per second per IP is recommended to preserve server resources without blocking legitimate SEO analysis, in line with best practices published on the official JoeDog support site.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.