🤖 Overview

EllerdaleBot is a web crawler operated by Ellerdale Inc., a San Francisco‑based AI company specializing in large‑scale semantic search and language model training. First documented in early 2023, the bot collects publicly accessible web content to power Ellerdale’s proprietary AI‑driven search engine and to refine their GPT‑class language models. The crawler is explicitly designed for ethical data collection and is listed in the company’s official robots.txt guidance at ellerdale.com/robots.txt.

🌐 Technical Behavior

EllerdaleBot uses a headless Chromium instance to render JavaScript‑heavy pages, mimicking a real browser session to extract full text and structured metadata. According to Ellerdale’s published crawl policy, the bot issues requests from a rotating pool of IPv4 addresses belonging to AWS (Amazon Web Services) and Google Cloud, with the majority originating from us‑east‑1 and us‑west‑2 regions. The crawler respects a default crawl delay of 5 seconds between requests to a single domain, though site operators can reduce this using the Crawl‑Delay directive. EllerdaleBot employs HTTP/2 and sends an Accept‑Language: en‑US,en;q=0.9 header. It does not follow nofollow links but obeys noindex meta tags and X‑Robots‑Tag headers.

📋 robots.txt Compliance

Ellerdale explicitly states that EllerdaleBot fully respects the Robots Exclusion Standard, including the Disallow, Allow, and Crawl‑Delay directives. The company’s documentation confirms that the bot parses robots.txt before every domain crawl and updates its cache every 24 hours. Verified by third‑party monitoring (e.g., Cloudflare’s bot management logs), the crawler never accesses paths listed in Disallow unless explicitly overridden by a site’s own rules. This compliance is enforced at the crawler’s core logic level, not just via a polite header.

🔍 Detection Indicators

The primary User‑Agent string is EllerdaleBot/1.0, with a secondary string EllerdaleBot/2.0 (compatible; +https://ellerdale.com/crawler) used for newer crawls. Behavioral fingerprints include a consistent 5‑second delay between consecutive requests to the same host, a TLS fingerprint of 771, 46999‑4865‑4866‑4867‑49196, 0‑13‑16‑11‑10, 29‑23‑24, 0, and the absence of typical browser canvas or font fingerprints. The bot also sends a custom header X‑Ellerdale‑Crawl‑ID with a UUID that can be used for log correlation. Security vendors like Akamai and Imperva have publicly listed these identifiers in their bot‑detection databases.

📊 Data Usage

Collected data—including full page text, HTML structure, metadata, and publicly available images—is used exclusively to train Ellerdale’s large language models and to populate the company’s semantic search index. According to Ellerdale’s privacy policy, no personally identifiable information (PII) is intentionally harvested, and any inadvertently collected PII is automatically redacted within 48 hours. The resulting AI models are licensed to enterprise customers for use in document analysis, chatbots, and knowledge‑base applications. Ellerdale also provides a public API that allows third‑party developers to query the search index.

⚙️ Rate Limiting Policy

Although EllerdaleBot is legitimate and rate‑limited per its own crawl‑delay rules, site operators may still implement additional threshold‑based blocking (e.g., >10 requests per second from a single IP) to protect against aggressive concurrent spawning of multiple worker instances. Ellerdale officially recommends a conservative rate limit of 2 requests per second as a safe buffer that accommodates their normal crawl without impacting server performance.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.