🤖 Overview

Dragonfly is a web crawler operated by Dragonfly Data Inc., a technology company specializing in large-scale web data extraction for AI model training and research purposes. First indexed in early 2023, Dragonfly systematically collects publicly accessible web content to feed into its proprietary dataset used for natural language processing and machine learning training.

🌐 Technical Behavior

Dragonfly performs a broad crawl across the internet, typically starting with high-traffic domains and following links recursively. Requests are made using HTTP/1.1 and HTTP/2, with a default crawl rate of up to 10 requests per second per domain, though this can vary based on server response times. The crawler identifies itself via the User-Agent string Dragonfly/1.0 and resolves to IP addresses primarily from AWS and Google Cloud ranges, documented in published IP feeds. It supports both IPv4 and IPv6 and uses standard HTTP headers such as Accept-Encoding and Connection. Crawl sessions are persistent, and the bot respects Cache-Control and Last-Modified headers to reduce server load.

📋 robots.txt Compliance

According to official documentation at dragonf.ly/robots, Dragonfly fully honors robots.txt directives, including Disallow rules, Crawl-Delay, and Sitemap references. It checks robots.txt at the start of each crawl session and re-evaluates periodically based on the TTL header. There is no evidence of intentional disregard for these directives in any published audit or community report.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Dragonfly/1.0; +https://dragonf.ly/bot). Additional fingerprints include a custom HTTP header X-Dragonfly-Crawl: yes and a distinctive TLS cipher suite preference that favors ECDHE-RSA-AES256-GCM-SHA384. The bot's IP ranges are published in a JSON feed at dragonf.ly/ipranges.json and are used by network administrators to apply rate limits.

📊 Data Usage

Collected data is used exclusively for training Dragonfly's proprietary large language models and for academic research partnerships under strict data-use agreements. The dataset is not publicly distributed and is subject to internal privacy and ethical review. Dragonfly also provides a data usage page for website owners to opt out via a dedicated form, as required by privacy regulations.

⚙️ Rate Limiting Policy

Dragonfly is rate-limited because its high crawl volume can impact server performance on smaller sites. Web administrators are advised to implement threshold-based blocking (e.g., 100 requests per minute) to prevent resource exhaustion while allowing the bot's legitimate functionality, as recommended in the official operator guidelines.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.