🤖 Overview

FacebookBot is a legitimate web crawler operated by Meta (formerly Facebook Inc.), first documented around 2010, whose primary purpose is to fetch and index publicly accessible web content so that links shared on Facebook, Messenger, Instagram, and WhatsApp can generate accurate link previews, rich sharing cards, and social metadata. The bot is also used to power Meta’s on-platform search and to improve content recommendation algorithms. According to Meta’s official developer documentation and the facebookexternalhit User‑Agent string release notes, the crawler behaves identically to a standard HTTP user‑agent and adheres to industry‑standard scraping etiquette.

🌐 Technical Behavior

FacebookBot crawls from IP ranges published by Meta in their AS32934 (Facebook) and AS54113 (Fastly CDN edge) netblocks, with specific subnets documented in Meta’s ip-ranges.txt file available at https://developers.facebook.com/docs/sharing/webmasters/crawler. The bot makes HTTP GET requests (both plain HTTP/1.1 and HTTPS/1.1) with a default fetching rate that can reach several hundred requests per minute from a single IP when a high‑traffic URL is shared widely, though Meta states it respects Retry-After headers and response latencies. It follows redirects (301, 302) with a maximum hop count of 20 and caches fetched pages for up to 30 days in Meta’s internal CDN. The crawler does not execute JavaScript or submit forms; it exclusively parses static HTML and tags such as og:title, og:image, og:description, and standard link rel="canonical". Request headers commonly include User‑Agent: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) and Accept-Language: en‑US,en;q=0.5, with no Referer field.

📋 robots.txt Compliance

Meta explicitly states that FacebookBot honors robots.txt Disallow directives and provides a dedicated User‑agent: facebookexternalhit rule in its official webmaster guide. Evidence from both the 2010 original documentation and a 2023 update confirms that the crawler will not fetch URLs explicitly blocked via Disallow, nor will it override X‑Robots‑Tag headers. However, site operators should note that if a Disallow directive is absent, the bot may aggressively re‑crawl pages that are shared often, so explicit rules are recommended for sensitive paths.

🔍 Detection Indicators

The primary identifying User‑Agent strings are facebookexternalhit/1.1 and the legacy Facebot (with variant facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)). A secondary detection method is the presence of the header Via: 1.1 facebook or X‑Forwarded‑For from Meta’s IP ranges. Behavioral fingerprints include a very low request interval when a single URL is freshly shared (within seconds of the share event) and a strict adherence to Open Graph protocol extraction.

📊 Data Usage

Collected data—page titles, descriptions, images, and favicons—is used exclusively to generate link previews for Meta’s social platforms and to populate the shared link metadata cache. No content is used for AI training or sold to third parties, as confirmed by Meta’s privacy policy and the 2023 Facebook External Hit FAQ. The bot does not store full page text beyond the metadata fields.

⚙️ Rate Limiting Policy

Because FacebookBot can fire hundreds of concurrent requests when a popular link is shared across Meta’s network, it is rate‑limited to protect origin server reliability. A sensible policy is to allow at least 20 requests per second per IP from Meta’s netblocks and to return a 429 Too Many Requests after exceeding 60 requests per IP per minute, with a Retry‑After header of 10 seconds—this prevents overload while still enabling rich‑link functionality.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.