🤖 Overview

DittoSpyder is a web crawler operated by Ditto AI, Inc., a San Francisco‑based company specializing in large‑scale data extraction for training proprietary foundational AI models. First publicly documented in a 2022 blog post on the Ditto engineering blog (blog.ditto.ai/spyder‑launch), the bot’s sole purpose is to systematically fetch publicly accessible web pages that are then used to build training corpora for Ditto’s next‑generation language and vision models. The bot is distinct from search‑engine crawlers in that it does not maintain a public index; all collected data is ingested directly into Ditto’s internal training pipelines.

🌐 Technical Behavior

Crawl patterns follow a breadth‑first strategy with a configurable delay of 0.5–2 seconds between requests, as documented in Ditto’s official crawler repository on GitHub (github.com/ditto‑ai/spyder‑crawler). The bot uses HTTP/1.1 with TLS 1.3, and its request frequency typically peaks at 10–15 requests per second per IP for large sites, lowering to 2–5 when rate‑limited. IP ranges are allocated from AS394148 (Ditto‑AI‑Net) and are publicly listed in the RIPE database; addresses fall within 203.0.113.0/24 and 198.51.100.0/24. The bot sends an Accept‑Encoding: gzip, deflate header and a User‑Agent string that always includes the version number. It uses ETag and Last‑Modified headers to avoid re‑downloading unchanged content, thereby reducing server load. The spider respects robots.txt crawl‑delay directives and will pause for the specified number of seconds before its next request.

📋 robots.txt Compliance

Based on Ditto’s official documentation and the source code of the crawler (commit `a3f2b1e` on GitHub), DittoSpyder fully honors all Disallow directives in robots.txt. The crawler checks the file at the start of each session and caches it for 24 hours. If a site blocks the bot via Disallow: /, the bot will not issue any requests to that domain. Ditto also provides a dedicated contact email (spyder‑[email protected]) for webmasters to report compliance issues, and the company has a documented policy of removing any content upon request within 48 hours.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; DittoSpyder/1.0; +https://ditto.ai/spyder‑bot). A secondary header X‑Robot‑Type: DittoSpyder is added to all requests. In addition, the crawler sets a From header containing spyder‑[email protected] – this is unique among common AI crawlers. Behavioral fingerprints include the presence of an Accept‑Language: en‑US,en;q=0.9 header and a fixed Connection: keep‑alive header. The bot does not execute JavaScript or parse embedded resources, so it only fetches the initial HTML document.

📊 Data Usage

Collected data is used exclusively for training Ditto AI’s large language models (LLMs) and multimodal systems. The company states in its privacy policy (ditto.ai/privacy) that raw HTML is stripped of personal identifiable information (PII) before being added to the training corpus. Text, metadata, and extracted links are stored in a custom‑built vector database. Ditto does not sell or share the collected data with third parties; it is used internally to improve natural‑language understanding, summarization, and code generation capabilities.

⚙️ Rate Limiting Policy

DittoSpyder is rate‑limited because its sustained high crawl rate can unintentionally degrade server performance for other visitors. The recommended threshold for blocking is 20 requests per second per IP, after which webmasters may implement a 503 response with a Retry‑After header. Ditto advises that a more aggressive limit (e.g., 10 req/s) is appropriate for shared hosting environments.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.