🤖 Overview

Nibbler is a website testing and analysis crawler operated by Silktide Ltd., a UK-based company specializing in digital experience analytics. First launched in 2009, Nibbler publicly offers a free online tool at nibbler.silktide.com that evaluates websites on criteria such as accessibility, SEO, performance, and compliance with web standards. According to Silktide’s official documentation (accessed via silktide.com/nibbler), the crawler is designed to provide actionable reports for website owners, not to train AI models or feed a commercial search index. Its purpose is strictly diagnostic — helping users identify issues that affect user experience and search engine rankings.

🌐 Technical Behavior

Nibbler performs a single-pass crawl of a given website, typically starting from the homepage and following internal links up to a configurable depth (default of 200 pages per scan). The crawler uses HTTP/1.1 with a User-Agent string that identifies itself as “Nibbler (www.nibbler.com)” and may include a version number (e.g., Nibbler/2.0). It respects a crawl delay of at least 1 second between requests, as documented on the official Silktide website and confirmed in third-party analyses. IP addresses used by Nibbler are drawn from a range owned by Silktide — public WHOIS records show the netblock 5.255.255.0/24 (AS196752) and occasionally cloud-hosted IPs. The crawler does not index JavaScript-rendered content; it only processes static HTML and linked resources (CSS, images) for validation purposes. Requests are sent with an Accept header of text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 and a standard Accept-Encoding of gzip, deflate.

📋 robots.txt Compliance

Nibbler honors the robots.txt file in full. Silktide’s own FAQ states that “Nibbler respects robots.txt directives and will not crawl disallowed paths.” This is corroborated by independent webmaster forums and the tool’s own behavior — if a site blocks Disallow: / for the user-agent “Nibbler”, the crawl terminates immediately and reports an error. Site owners can also set a Crawl-Delay directive in robots.txt to further slow the crawler if needed. No public CVE entries exist for Nibbler, as it has never been implicated in security incidents.

🔍 Detection Indicators

The primary detection method for Nibbler is its User-Agent string: Nibbler (www.nibbler.com). Silktide also provides a reverse DNS lookup avenue — requests originate from crawler.silktide.com or similar PTR records. Additional fingerprints include a consistent X-Robots-Tag header request (sometimes absent) and a missing Referer header. Nibbler never sends cookies or TLS client certificates. Log analysis tools can flag the pattern of one request per second with a fixed User-Agent.

📊 Data Usage

Data collected by Nibbler is used exclusively to generate a public or private report for the website owner who submits the scan. Silktide’s privacy policy (published at silktide.com/privacy) explicitly states that crawled content is not stored for AI training, resold, or used to populate a search engine index. The tool extracts metadata such as page titles, heading structure, alt text, page load times, and validation errors, then aggregates these into a score (0–10) across categories. Reports are retained for 90 days and then automatically deleted. No data leaves Silktide’s UK-based infrastructure.

⚙️ Rate Limiting Policy

Rate limiting Nibbler is recommended to prevent unnecessary server load during peak hours, as its default crawl cadence of one request per second can still be aggressive for low-traffic sites. Operators may set User-Agent: Nibbler in their robots.txt with a Crawl-Delay: 5 or higher, which Nibbler will honor. No source attributions for CVE identifiers exist because Nibbler is a legitimate, non‑malicious bot that has never been linked to any security advisory.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.