🤖 Overview

facebookexternalhit is a web crawler operated by Meta Platforms, Inc. (formerly Facebook) that fetches URLs shared on Facebook’s social network to generate link previews (Open Graph cards) and to populate the Facebook Sharing Debugger tool. First identified in public server logs around 2010, its primary purpose is to retrieve page metadata—titles, descriptions, images, and video information—so that when a user posts a link on Facebook, Messenger, or Instagram, the platform can display a rich, interactive preview. The bot also supports the Instant Articles and Facebook News Feed indexing pipeline.

🌐 Technical Behavior

The bot typically makes requests with a User-Agent string of facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) or variations like facebookexternalhit/1.2. It sends an additional header Via: 1.1 facebookexternalhit in some cases. According to Meta’s official developer documentation at developers.facebook.com, the crawler operates from IP ranges in the AS32934 (Facebook’s autonomous system), with public IP blocks such as 69.171.224.0/20, 31.13.24.0/21, 66.220.144.0/20, and others. The bot makes a single request per URL share, but for sites with heavy social sharing it may request the same URL multiple times within short intervals (often seconds) due to multiple users sharing the link. It uses HTTP/1.1 and HTTP/2 and supports gzip compression. The crawler does not execute JavaScript or load external resources; it only fetches raw HTML to parse Open Graph tags and structured data.

📋 robots.txt Compliance

Meta states that facebookexternalhit respects the robots.txt directives as documented on their crawler page. Specifically, it will not crawl URLs blocked by Disallow rules for the user-agent facebookexternalhit or the wildcard *. There is no evidence of intentional bypassing of robots.txt, and Meta advises webmasters to use robots.txt to block the bot if previews are not desired, though doing so will also break Facebook sharing functionality.

🔍 Detection Indicators

The primary identifier is the User-Agent string facebookexternalhit/1.1 (or /1.2). Additional fingerprints include the X-Purpose: preview header in some versions and a request pattern featuring rapid fire requests for a single URL within a few seconds when multiple users share the same link. Reverse DNS resolves to *.facebook.com or *.fbsv.net. The bot also includes a From: [email protected] header in certain cases.

📊 Data Usage

Collected content—specifically Open Graph metadata, titles, descriptions, and image URLs—is used solely for generating link previews on Facebook, Messenger, and Instagram. The bot does not use the data for advertising profiling, AI training, or long-term storage beyond what is necessary for the preview (typically cached for a limited time). Meta’s privacy policy at facebook.com/privacy confirms that shared URLs are processed ephemerally for preview generation.

⚙️ Rate Limiting Policy

This bot is rate-limited because its request frequency can spike unpredictably during viral sharing events, causing load spikes on small servers. A threshold-based block (e.g., >10 requests per second from a single IP) is recommended to protect server resources without breaking legitimate functionality, as the bot will retry with exponential backoff.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.