🤖 Overview

The opendirme-credhunt crawler is a specialized web agent developed by independent security researchers associated with the OpenDirMe project on GitHub. Its primary purpose is to systematically crawl public-facing web servers for openly accessible directories and files, specifically targeting credential-related data such as configuration files, backup archives, and plaintext passwords. This bot feeds its findings into a vulnerability disclosure pipeline that notifies site owners of exposed sensitive information, aligning with responsible disclosure practices.

🌐 Technical Behavior

Operates over HTTP/1.1 and HTTPS, initiating requests with a default concurrency of 10 threads and a configurable delay of 2–5 seconds between requests to avoid overwhelming target servers. It uses a breadth-first crawl strategy, first scanning for common directory paths like /backup/, /config/, and /.git/ before probing deeper. The crawler employs a custom DNS resolver to bypass basic IP restrictions and rotates a pool of approximately 50 residential IP addresses sourced from cloud providers (e.g., AWS, DigitalOcean). Request headers mimic a modern Chrome browser (User-Agent, Accept-Language, etc.) but include a custom X-Crawler-Type: opendirme-credhunt fingerprint for identification. Traffic peaks during non-business hours (UTC 02:00–06:00) to reduce operational impact.

📋 robots.txt Compliance

According to the official OpenDirMe documentation (https://github.com/opendirme/opendirme), the crawler fully respects robots.txt Disallow directives. It caches the robots.txt file for each domain and re‑checks it every hour. However, it does not honor meta robots tags in HTML because it operates primarily on directory listings rather than page content.

🔍 Detection Indicators

The primary User‑Agent string is opendirme‑credhunt/1.0 (Mozilla/5.0 compatible). Secondary variants include opendirme‑credhunt/2.0 and OpenDirMe‑Bot/1.0. It also sends a unique HTTP header X‑Crawler‑Variant: credential‑hunt. Behavioral fingerprints include sequential directory probing with 404‑driven back-off and a consistent request interval of 3 seconds ± 0.5 seconds.

📊 Data Usage

Collected credential files are analyzed for severity (e.g., hardcoded passwords, database credentials) and reported to the domain owner via email or a vulnerability disclosure platform like HackerOne. The data is never stored publicly or used for any adversarial purpose; the project’s GitHub repository (https://github.com/opendirme/opendirme) explicitly states a non‑commercial, ethical intent. Historical crawl data is retained for 90 days for verification purposes only.

⚙️ Rate Limiting Policy

Because the crawler can generate thousands of requests across a target site in a short period if directory listings are large, it is rate‑limited to prevent accidental denial‑of‑service. Threshold‑based blocking (e.g., 200 requests per minute per IP) is recommended to allow the crawler to complete its research without disrupting normal user traffic.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.