🤖 Overview

crawler.feedback is a legitimate web crawler operated by Crawler Feedback Inc., a company specializing in website experience analytics and accessibility auditing. First identified in public User‑Agent logs in early 2022, its primary purpose is to collect publicly accessible web content to feed the Crawler Feedback Platform, which provides site owners with actionable insights on page performance, user experience issues, and compliance with WCAG and GDPR standards. The bot is explicitly designed to support webmasters, not to train generative AI models.

🌐 Technical Behavior

The crawler performs depth‑limited visits, typically crawling no more than two levels from the entry page per session, with an average delay of 500 milliseconds between requests. It supports both HTTP/1.1 and HTTP/2 and advertises Accept‑Encoding: gzip, deflate, br in its request headers. IP ranges are drawn from AWS EC2 (us‑east‑1, eu‑west‑1) and Google Cloud (us‑central1, europe‑west1), rotating every 50 requests to distribute load. The bot sends a Referer header set to https://crawler.feedback and always includes the X‑Crawler‑Feedback‑ID header for traceability. According to the official documentation at crawler.feedback/robots, it respects the Crawl‑Delay directive when present, and never attempts to access password‑protected or dynamically generated content paths by default.

📋 robots.txt Compliance

As stated in the robots.txt guidance published on the Crawler Feedback website, the bot fully honors Disallow directives and stops crawling any path declared off‑limits. It also checks for Allow directives when partial access is granted. Independent testing by the Webmaster World forum in March 2023 confirmed that the bot never ignored robots.txt blocks during a 90‑day observation period.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; crawler.feedback/2.0; +https://crawler.feedback/bot). A secondary string crawler‑feedback‑bot/1.0 is used for JavaScript rendering crawls. Behavioral fingerprints include a very low variance in request times (±20 ms) and the consistent inclusion of the X‑Crawler‑Feedback‑Version header. Log entries typically show the bot hitting a site exactly every 5–8 seconds without fail.

📊 Data Usage

Collected data is aggregated into the Crawler Feedback Dashboard to produce scores for accessibility (WCAG 2.1 AA), page load speed, mobile responsiveness, and broken link detection. Website owners can view anonymized feedback reports without any redistribution of raw content to third parties. The company’s privacy policy, available at crawler.feedback/privacy, explicitly states that no personal data is stored beyond the URL and timestamp of the crawl.

⚙️ Rate Limiting Policy

The bot is intentionally rate‑limited because even well‑behaved crawlers can inadvertently overload smaller sites. The recommended threshold for blocking is 100 requests per minute from a single IP, after which the bot backs off exponentially; exceeding 300 requests per minute triggers a permanent block as per the platform’s fair‑use policy documented at crawler.feedback/rate‑limiting.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.