🤖 Overview

fetch api request (commonly known as Facebook External Hit) is a legitimate web crawler operated by Meta Platforms, Inc. Its primary purpose is to fetch link previews, Open Graph metadata, and page content when users share URLs on Facebook, Messenger, and other Meta services. The data feeds into Facebook’s link-sharing infrastructure and is used to generate rich previews (title, description, image) in posts and messages.

🌐 Technical Behavior

This bot issues standard HTTP GET requests to URLs shared by users, typically requesting the HTML content and inspecting tags, Open Graph protocol tags, and schema.org markup. It follows redirects and may request linked resources such as images for preview thumbnails. The crawler originates from IP ranges within Meta’s ASN (AS32934), including subnets like 31.13.24.0/21 and 69.171.224.0/19, and uses a consistent request frequency of roughly one request per URL per share, though multiple shares of the same URL can trigger repeated fetches. It uses HTTP/1.1 and negotiates gzip compression. The bot respects cache-control headers and may use If-Modified-Since to reduce load.

📋 robots.txt Compliance

fetch api request fully honors robots.txt Disallow directives. Meta explicitly documents that the crawler obeys standard exclusion rules, and multiple security analyses confirm that it does not bypass robots.txt restrictions. Site owners can block the bot entirely by denying access to the user-agent facebookexternalhit in their robots.txt file.

🔍 Detection Indicators

The primary User-Agent string is facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php). Occasionally, it may present as Mozilla/5.0 (compatible; FacebookExternalHit/1.1). The bot does not include typical browser headers like Accept-Language or Referer, and its IPs always resolve to *.facebook.com or *.fbcdn.net. The request path often includes a ?__tn__ query parameter used for internal tracking.

📊 Data Usage

Collected metadata is used exclusively for generating link previews on Meta platforms: the title, description, image, and site name are cached and displayed when a user shares a link. No full page content is stored; only extracted metadata is retained for performance. The data is not used for AI training or advertising targeting, as stated in Meta’s official documentation.

⚙️ Rate Limiting Policy

Because the bot can re-crawl the same URL multiple times (e.g., when many users share it), it is rate-limited to prevent excessive server load. Thresholds such as 10 requests per second per IP from a single source are recommended; blocking is reserved for abusive behavior, not routine crawling, as the bot is a legitimate part of Meta’s link-sharing ecosystem.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.