🤖 Overview

The facebookscraper refers to the suite of web crawlers operated by Meta Platforms, Inc. (formerly Facebook, Inc.) to fetch content from external websites for generating link previews, sharing metadata, and populating social graph information. This bot is documented under the official user agent names facebookexternalhit and Facebot, as described on Meta’s developer documentation at https://developers.facebook.com/docs/sharing/webmasters/crawler. Its primary purpose is to retrieve Open Graph (OG) tags, schema.org markup, and HTML meta-data from URLs shared on Facebook, Messenger, and Instagram, enabling rich previews (e.g., title, description, image, video) without storing full page content for long-term indexing.

🌐 Technical Behavior

The crawler operates from IP ranges owned by Meta, documented in their official AS32934 (Facebook, Inc.) and AS54113 (Fastly CDN) announcements. Typical request patterns show bursts of activity when users share links on social platforms—each share triggers a single crawl of the shared URL, with subsequent crawls if the page’s OG tags change or are re-fetched via the Debugger tool. The bot sends HTTP GET requests with a default concurrency of roughly 10–20 simultaneous connections per IP, and respects Cache-Control headers (e.g., max-age) to reduce re-crawl frequency. Historically, Facebook’s crawler has been reported to ignore If-Modified-Since headers in some versions, but as of 2024 updates it honors 304 Not Modified responses. The user agent string includes versioning, e.g., facebookexternalhit/1.1 or Facebot, and the bot identifies itself with the X-Purpose header set to “preview”.

📋 robots.txt Compliance

Meta explicitly states on its developer portal that facebookexternalhit and Facebot obey standard robots.txt directives when fetching pages for link sharing. However, there is a known nuance: the crawler respects Disallow rules, but it may still attempt to fetch a URL even if a Disallow is present for deeper paths if the share originates from a user—Meta’s documentation advises that you can block the crawler entirely by disallowing the specific user agent strings. Public evidence from forums indicates that many sites effectively rate-limit or block this bot via robots.txt without negative impact on social sharing, as the bot will return a fallback if blocked.

🔍 Detection Indicators

Identifying the Facebook crawler in server access logs is straightforward via two principal user-agent strings: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) and Facebot. Additional signals include the X-Purpose: preview header, and frequent requests from IP ranges within AS32934 (e.g., 31.13.24.0/21 for Facebook’s US datacenters) or AS54113 for CDN‑cached previews. A behavioral fingerprint is its pattern of requesting only the first few kilobytes of the page—typically up to 512 KB—to parse OG tags, then disconnecting.

📊 Data Usage

The collected data—specifically Open Graph, Twitter Card, and schema.org metadata—is used solely for generating real‑time link previews in Meta’s social products. No full page content is permanently stored; the metadata is cached temporarily (typically up to 24 hours) for display when the shared link is clicked. Meta does not use this data for training AI models or long‑term analytics, per its Privacy Policy (see https://www.facebook.com/privacy/policy). The crawler also fetches image assets (OG images) and caches them for thumbnail rendering.

⚙️ Rate Limiting Policy

Because the Facebook crawler can trigger bursts of traffic from multiple users sharing the same URL simultaneously, web administrators often rate‑limit it using throttling rules that limit requests per IP per second (e.g., 5 req/s). Such blocking is justified to prevent server overload while still allowing legitimate previews; Meta recommends using Cache-Control headers rather than blocking outright to avoid degraded social sharing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.