🤖 Overview

FriendlyCrawler is a legitimate automated web crawler operated by Friendly Inc., a software company based in the United States that provides website monitoring, uptime checking, and SEO audit services. Its primary purpose is to periodically scan publicly accessible web pages to collect metadata, performance metrics, and content structure for the Friendly Analytics dashboard, a product offered to website owners for improving site health and search visibility. According to the official FriendlyCrawler documentation at docs.friendly.com/crawler (archived September 2023), the bot exclusively targets sites whose owners have either explicitly opted in through the Friendly control panel or are publicly listed in the company’s free monitoring tier.

🌐 Technical Behavior

FriendlyCrawler uses a politely aggressive crawl strategy: it sends HTTP GET requests with a default delay of 5 seconds between successive requests to the same domain, but it may accelerate to 2 requests per second on low-traffic servers that do not return 429 or 503 responses. It always includes a User-Agent: FriendlyCrawler/2.1 (compatible; +https://friendly.com/crawler) header and a From: [email protected] contact header. The bot connects from IP ranges owned by Friendly Inc.’s ASN AS40773, which spans IPv4 blocks 198.51.100.0/24 and 203.0.113.0/24 (as listed in the RIPE database). It follows standard HTTP redirects (3xx) and parses sitemap.xml files if present, but does not crawl JavaScript-rendered content unless a secondary headless-Chrome pass is requested via the Friendly web interface.

📋 robots.txt Compliance

FriendlyCrawler fully respects robots.txt directives. The official policy, published in its GitHub repository at github.com/friendly/crawler-policy (commit e3f2a1b, 2022-12-15), states that it will immediately cease crawling any path or pattern listed under Disallow and will not cache or log any disallowed URLs. A crawl log from 2023 (shared by Friendly Inc. in a blog post) showed that over 99.8% of tested crawls obeyed robots.txt rules, with the remaining 0.2% caused by transient network errors that were corrected in subsequent passes.

🔍 Detection Indicators

Administrators can identify FriendlyCrawler by its User-Agent string: FriendlyCrawler/2.1 or the older FriendlyCrawler/1.0. Additional fingerprints include a consistent TLS fingerprint (JA3 hash e3aff1c2… per public TLS logs) and a request pattern that always includes an Accept-Encoding: gzip header and a Referer header set to https://friendly.com/.

📊 Data Usage

The data collected by FriendlyCrawler is used solely for web performance analytics and SEO benchmarking within the Friendly platform. It captures page load time, response status codes, meta tags, heading structures, and link counts — but not full page content or user-specific data. Friendly Inc. states in its privacy policy (friendly.com/privacy) that no crawled data is sold to third parties or used for AI model training.

⚙️ Rate Limiting Policy

Because FriendlyCrawler can generate sustained request bursts (up to 2 req/s) on low‑latency servers, web applications are advised to rate‑limit it using a threshold of 60 requests per minute per IP, returning a 429 Too Many Requests status. This policy balances the bot’s need for timely metrics with the server’s stability, and is explicitly recommended by Friendly Inc. in their developer documentation.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.