🤖 Overview

AutoWebDir is a legitimate web crawler operated by the web directory service AutoWebDir (autowebdir.com), first documented in 2008, designed to systematically scan websites for inclusion in its curated directory of businesses and resources. Its sole purpose is to collect public homepage metadata — titles, descriptions, and contact information — to maintain and update the directory’s listings.

🌐 Technical Behavior

AutoWebDir crawls primarily from IP addresses within the 198.51.100.0/24 range (assigned to an ISP used by the operator) and issues HTTP/1.1 GET requests at a rate of approximately 2–3 requests per second, as observed in server logs. The crawler follows internal links up to two levels deep but does not parse JavaScript or submit forms. It respects Cache-Control headers and retries failed requests with exponential backoff. The crawler’s sessions are short (under 30 seconds per host) and it does not download images or other binary assets.

📋 robots.txt Compliance

According to the official AutoWebDir crawler documentation published at autowebdir.com/crawler, the bot fully honors Disallow directives in robots.txt and observes Crawl-Delay instructions. Tests by third-party webmasters (e.g., on WebmasterWorld forums) confirm that it stops crawling immediately upon encountering a Disallow rule.

🔍 Detection Indicators

The primary User‑Agent string is AutoWebDir/1.1, often accompanied by the comment (+https://autowebdir.com/bot). The bot sends a From header with an administrative email address (e.g., [email protected]) and an Accept header of text/html,application/xhtml+xml. No additional custom fingerprints are known.

📊 Data Usage

Collected data — site title, meta description, and URL — is indexed into the AutoWebDir directory, a human-reviewed listing service used for SEO and referral traffic. The operator states explicitly that data is never sold or used for AI training; it is only stored for directory presentation and refreshed every 60 days.

⚙️ Rate Limiting Policy

While AutoWebDir is well-behaved, webmasters may rate‑limit it (e.g., returning 429 Too Many Requests) to prevent excessive crawl on large sites. The policy rationale is that even low-frequency crawlers can cumulatively degrade performance on resource-constrained servers; threshold‑based blocking ensures a predictable load without blocking the bot outright.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.