🤖 Overview

scan.lol is a web crawler operated by an independent security research collective, first observed in public logs around 2022. Its primary purpose is to systematically scan the public internet for misconfigured or exposed assets—such as open .git directories, unprotected environment files (.env), and accessible admin panels—and to index those findings on the scan.lol website for disclosure coordination with affected parties. The project is documented at https://scan.lol and is explicitly described as a “non‑malicious security reconnaissance” tool aimed at reducing the attack surface of web applications by surfacing common misconfigurations.

🌐 Technical Behavior

The crawler performs host‑based scans by enumerating common paths and file names (e.g., /.git/config, /admin/, .env) over HTTP and HTTPS, typically issuing bursts of requests at a rate of 10–50 requests per second per target IP. It sources its scanning infrastructure from a rotating pool of cloud providers including DigitalOcean, Linode, and Hetzner, with IP ranges that change frequently. According to the project’s GitHub repository (https://github.com/scanlol), the crawler uses a multi‑threaded asynchronous model and prioritizes scanning of IPv4 address blocks that are publicly announced on BGP tables. It does not attempt to exploit vulnerabilities; it only reports the existence of accessible resources.

📋 robots.txt Compliance

Per the scan.lol documentation, the crawler is designed to honor robots.txt directives by default, provided the file is served with a 200 OK status and contains a clear Disallow rule targeting the scan.lol User‑Agent string. However, due to the high scan volume and the occasional absence of a robots.txt file on target hosts, web administrators have reported instances where the crawler continued to scan paths that were not explicitly disallowed. The project maintainers encourage site owners to add User-agent: scan.lol and Disallow: / if they wish to opt out completely.

🔍 Detection Indicators

The primary identifying User‑Agent string is scan.lol (sometimes with a version suffix like scan.lol/1.0). The crawler also sets a custom HTTP header X-Scanner: scan.lol on every request. Behavioral fingerprints include sequential requests for a large number of high‑risk paths (e.g., /wp-admin/, /backup/, /config.php) within a few seconds, originating from a single IP address that has no referrer or cookie. The pattern is easily distinguishable from search engine crawlers because of the spike in 403/404 responses for non‑existent resource paths.

📊 Data Usage

Collected data—specifically URLs and HTTP response status codes indicating exposed resources—is stored in a public database at scan.lol and is used solely for security research and responsible disclosure. The project does not use the data for AI training, advertising, or any commercial purpose. According to the site’s FAQ, findings are reported to the affected domain owners via automated email notifications when contact information is available, and the data is periodically cleaned of stale entries.

⚙️ Rate Limiting Policy

Because scan.lol can generate a high volume of requests in a short time window—often triggering Web Application Firewalls (WAFs) and causing moderate load on smaller servers—it is prudent to rate‑limit the bot to prevent resource exhaustion. A threshold of 100 requests per minute per IP is a commonly recommended practice to balance the crawler’s legitimate scanning purpose with the operational stability of the target application.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.