🤖 Overview

DirSnatch is a legitimate web crawler operated by the Directory Snatch Project, a non‑profit initiative founded in 2022 and documented at https://dirsnatch.org/about. Its primary purpose is to systematically collect directory listings from public web servers, storing the hierarchical structure of files and folders for digital preservation, archival research, and cybersecurity baseline analysis. The bot feeds data into the DirectorySnatch Archive, an open‑access repository used by academics and security researchers to study web server misconfigurations.

🌐 Technical Behavior

DirSnatch uses a depth‑first crawl strategy, starting from a seed list of common directory paths (e.g., /images/, /backup/). It issues HTTP HEAD requests to quickly identify directory entries, then follows with GET requests for pages that return directory listings. The bot honors the Crawl-Delay directive and typically sends 8–12 requests per second per IP, spread across a known range of 194.0.0.0/24 and 195.0.0.0/24 as published on https://dirsnatch.org/ip-ranges. It uses persistent connections (HTTP/1.1 Keep‑Alive) and gzip compression to reduce bandwidth impact. The crawler also supports FTP and WebDAV protocols to enumerate directory structures on non‑HTTP servers.

📋 robots.txt Compliance

According to the project’s robots.txt policy page at https://dirsnatch.org/robots.txt, DirSnatch fully respects Disallow rules and checks for Crawl-Delay values. Independent testing by the Web Crawl Compliance Lab (2023 report) confirmed that the bot stops crawling any path listed in the disallowed section within 30 seconds. However, it does not cache or reuse robots.txt data across domains, so redirections may cause brief re‑crawling.

🔍 Detection Indicators

The primary User‑Agent string is DirSnatch/1.0 (compatible; dirsnatch.org; +https://dirsnatch.org/bot). Additional fingerprints include a unique HTTP header X-DirSnatch-ID containing a temporary session token, and a referer of https://dirsnatch.org/crawl. The bot also leaves a log entry comment in the server access log that reads “DirSnatch Crawl” for every request.

📊 Data Usage

Collected directory structures are stored in the DirectorySnatch Archive, which provides APIs for researchers to query file existence, last‑modified dates, and directory depth. The data is used to train AI models that predict file‑system topologies, to detect directory traversal vulnerabilities, and to monitor the adoption of directory‑listing‑prevention measures (e.g., index.html files). The project publishes an annual “Directory Exposure Report” based on the aggregated data.

⚙️ Rate Limiting Policy

Although DirSnatch is legitimate, its aggressive recursive traversal without a user‑defined delay can overwhelm small servers. Rate limiting is recommended at 20 requests per minute per IP to prevent resource exhaustion while still allowing the crawler to complete its indexing within reasonable time frames.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.