🤖 Overview

The meta-externalfetcher is a web crawler operated by Meta Platforms, Inc. (formerly Facebook) and publicly documented since early 2024. Its primary purpose is to collect publicly available web content for training and improving Meta’s large language models (LLMs), including the LLaMA series, and to enhance other AI‑powered products such as content recommendation and translation systems. The bot is explicitly described on Meta’s official AI crawler page (https://about.meta.com/ai/crawler/) and is part of a broader ecosystem of external data fetchers used for research and model development.

🌐 Technical Behavior

According to Meta’s documentation and observed patterns, meta-externalfetcher issues HTTP GET requests at a moderate rate, typically between 1 and 5 requests per second per IP, though bursts may occur during initial crawling of a domain. The bot respects standard HTTP headers and uses IPv4 and IPv6 ranges announced via Meta’s ASN (AS32934); specific prefixes are listed in the https://developers.facebook.com/docs/sharing/crawler/ page. It requests both HTML and linked resources (e.g., images, PDFs) but does not execute JavaScript. Requests originate from user‑agent strings ending with `meta-externalfetcher/1.0` and include the `From` header with a contact email ([email protected]). The crawler adheres to the `Accept‑Language` header and sends a `User‑Agent` containing “compatible” keywords for compatibility. No IP ranges are publicly committed to a PTR record, but reverse DNS lookups have shown “mfetcher.xx.facebook.com” patterns.

📋 robots.txt Compliance

Meta explicitly states that meta-externalfetcher respects the robots.txt exclusion protocol. The bot checks the file before each crawl session and obeys `Disallow` directives. Official guidance on https://developers.facebook.com/docs/sharing/crawler/ advises webmasters to use the `User‑agent: meta-externalfetcher` line in their robots.txt to control access. No reports of persistent violations have been documented in security advisories or community forums. However, like many AI crawlers, it may re‑crawl a page after changes, so a stale robots.txt could be read later.

🔍 Detection Indicators

The definitive User‑Agent string is Mozilla/5.0 (compatible; meta-externalfetcher/1.0; +https://about.meta.com/ai/crawler/). Some secondary variants append platform information but always include `meta-externalfetcher`. Behavioral fingerprints include a consistent `From` header identical to the contact email, a default `Accept: text/html,application/xhtml+xml` preference, and a request frequency that rarely exceeds 10 requests per second per source IP. Log entries typically show the `X‑Forwarded‑For` header missing, as the crawler often uses direct connections. The `Via` header may contain “meta-externalfetcher” if proxies are involved.

📊 Data Usage

Collected data is used exclusively for AI training (LLaMA models), model evaluation, and improving Meta’s natural language understanding. According to Meta’s transparency page, the crawler does not collect personal identifiable information (PII) beyond what is publicly available on the web, and that data is anonymized or aggregated before training. The data may also feed internal research on content safety and ranking algorithms. Meta provides a web form for takedown requests for specific content (https://about.meta.com/ai/data‑opt‑out).

⚙️ Rate Limiting Policy

Because meta-externalfetcher can generate non‑trivial traffic, it is rate‑limited to protect server resources and maintain fair use. A threshold of 20 requests per second per IP is a common industry recommendation; exceeding this for more than 60 seconds may warrant temporary blocking. The policy rationale is to ensure the crawler does not degrade web application performance for real users while still allowing thorough data collection for AI research.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.