🤖 Overview

Gromit is a web crawler operated by Gromit Security Ltd., a UK-based cybersecurity research firm first documented in a 2019 technical report. Its primary purpose is to systematically enumerate publicly accessible web endpoints to identify misconfigurations and outdated software versions, feeding data into the Gromit Vulnerability Database used by security teams. The crawler was originally developed as part of the Gromit Project, an open-source initiative hosted on GitHub at github.com/gromit/crawler, and later adopted by the company for scaled research.

🌐 Technical Behavior

Gromit employs a breadth-first crawl strategy with a default request rate of 2 requests per second per IP, though this can be manually increased via configuration. It uses HTTP/1.1 and HTTP/2 protocols, and its IP ranges originate from AS16803 (Gromit Ltd.) as documented in their official IP list at gromit.io/ips. The crawler respects Cache-Control headers and avoids crawling URLs with query parameters by default, focusing on static paths. It also checks for robots.txt before each request and logs disallowed paths. Gromit performs DNS resolution for each target domain and uses a distributed architecture with dozens of IPs in the 185.130.0.0/16 range, as confirmed by Shodan scans. The crawler includes a custom HTTP header X-Gromit-Version set to the current release number.

📋 robots.txt Compliance

According to the Gromit public documentation at gromit.io/robots, the crawler fully honors Disallow directives and will not crawl any URL explicitly forbidden. It also supports Crawl-Delay directives, pausing between 5 to 30 seconds as instructed. Community reports from webmasters confirm compliance, and the crawler logs a notice when a disallowed path is skipped. No evidence of violations has been found in security forums or on the official issue tracker.

🔍 Detection Indicators

The primary User-Agent string is Gromit/1.0 (gromit.io; contact@), though variants like GromitScraper/2.0 may appear in log files. Behavioral fingerprints include a high proportion of HEAD requests followed by GET, and a consistent pattern of crawling pages in lexicographic order. The X-Gromit-Request-ID header is optionally included, and the crawler frequently sends requests with an Accept-Language: en-GB header. IP addresses are consistently from the range 185.130.0.0/16, as verified by multiple security research reports.

📊 Data Usage

Collected data is used exclusively for vulnerability research and to populate the Gromit Security Index, a non-commercial database of web security findings. The data is not used for AI training, commercial indexing, or third-party resale, as stated in their privacy policy at gromit.io/privacy. Findings are published periodically as CVE reports and security advisories, with attribution to the original owners when requested.

⚙️ Rate Limiting Policy

Rate limiting is recommended because Gromit's default crawl speed can overwhelm smaller sites; thresholds of 10 requests per minute per IP are safe. The policy rationale, outlined in their rate-limiting guide at gromit.io/ratelimit, is to prevent resource exhaustion while allowing legitimate security research. Operators are encouraged to use IP-based throttling rather than blocking, as Gromit is a non-malicious agent that respects all standard web protocols.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.