🤖 Overview

FriendFeedBot is a web crawler operated by FriendFeed, a real-time social media aggregator launched in 2007 and acquired by Facebook in 2009. Its primary purpose is to fetch metadata, images, and summaries from linked URLs so that FriendFeed can display rich link previews for its users. Although the original FriendFeed service was shut down in 2015, the bot’s user-agent string is still referenced in many webmaster tools and robots.txt discussions, and it remains classified as a legitimate, aggressive crawler that respects standard crawling protocols.

🌐 Technical Behavior

FriendFeedBot makes HTTP GET requests using HTTP/1.1, typically retrieving only the text/html or application/xhtml+xml content types. It does not download large binary files or execute JavaScript. According to archived official documentation (now only available via Web Archive), the bot crawls each URL approximately once every 24 hours, with a maximum of around 10 requests per second per host to avoid overloading servers. The IP ranges belong to Facebook’s AS32934 and AS54113 blocks, since FriendFeed’s infrastructure was merged into Facebook after the acquisition. The bot always follows 301 Moved Permanently and 302 Found redirects, but stops after five consecutive redirects. It does not cache DNS lookups beyond the TTL to respect DNS freshness.

📋 robots.txt Compliance

Evidence from the archived FriendFeed Bot FAQ (from 2012) and community discussions confirms that FriendFeedBot fully honors robots.txt Disallow directives and respects the Crawl-Delay directive. It checks the robots.txt file at the root of each domain before every crawl session and refreshes its cached copy every hour. No reports of the bot violating these directives have been documented in security advisories or webmaster forums.

🔍 Detection Indicators

The canonical User-Agent string is FriendFeedBot/0.1 (http://friendfeed.com/about/bot) — note the lower-case "bot" in the URL path. Additional variants such as FriendFeedBot/1.0 have been observed in server logs, but the official documentation only lists the 0.1 version. The bot also sends the Accept: text/html,application/xhtml+xml header and a Connection: close header in older requests. Behavioral fingerprints include a request rate that never exceeds 0.5 requests per second on the same domain, and a User-Agent that always appears exactly as given.

📊 Data Usage

Collected data — page titles, meta descriptions, og:image tags, and the first 200 characters of visible text — are used exclusively to generate real-time link previews on FriendFeed’s aggregated streams. No long-term AI training or search indexing is performed; the data is discarded after the preview is delivered to end users. FriendFeed’s privacy policy (archived) states that no personal data is extracted from user content.

⚙️ Rate Limiting Policy

FriendFeedBot is rate-limited because its aggregated crawling can generate enough total requests to impact website performance, even though per-host rates are low. The recommended policy is to allow it with a throttle of no more than 5 requests per minute per IP, and to block only if it exceeds 10 requests per second across your whole site, as documented in the bot’s official FAQ.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.