Skip to main content

Boteraser | Website and Server Security Solutions

metaeuro web search

Search Engine User-Agent: metaeuro-web-search

🤖 Overview

MetaEuro Web Search is a legitimate web crawler operated by Meta Platforms, Inc. (formerly Facebook) for indexing publicly accessible web content to power its European-focused web search product and to improve AI models used within Meta’s ecosystem. Officially documented in Meta’s crawler registry, this bot is distinct from Meta’s general-purpose crawlers and is specifically deployed to comply with European data regulations while supporting Meta’s search features on platforms like Facebook and Instagram.

🌐 Technical Behavior

The crawler sends HTTP requests with a standard user-agent string and employs both HTTP/1.1 and HTTP/2 protocols. According to Meta’s published infrastructure, it originates from IP ranges within Meta’s autonomous system AS32934, including subnets such as 69.171.224.0/19 and 31.13.24.0/21. It typically issues requests at a steady rate of 10–20 per second, though burst activity may occur during initial index updates. The crawler respects Crawl-Delay directives in robots.txt and supports conditional If-Modified-Since headers to reduce bandwidth usage. Reverse DNS lookups of its IP addresses resolve to hostnames under *.fbsv.com or *.facebook.com.

📋 robots.txt Compliance

Meta explicitly states in its official crawler documentation that MetaEuro Web Search honors all Disallow directives in robots.txt. The crawler’s behavior aligns with Meta’s published policy, which includes support for User-agent: MetaExternalAgent and User-agent: Meta-ExternalAgent lines. Failure to comply would contradict Meta’s own guidelines and has not been reported in any security advisories.

🔍 Detection Indicators

The primary User-Agent string is MetaExternalAgent/1.0 with additional variants like Meta-ExternalAgent/1.0 and MetaEuro/1.0. Behavioral fingerprints include consistent intervals between requests (typically 0.5–2 seconds) and a preference for crawled pages over non‑text content. The crawler does not alter Accept-Language headers unusually and sends standard User-Agent and From fields containing Meta’s contact information.

📊 Data Usage

Collected data feeds into Meta’s internal web search index, used to surface relevant web results within Facebook’s search bar and Instagram’s explore features. Additionally, a subset of crawled content is used to train Meta’s large language models (such as Llama) and improve content recommendation algorithms, as noted in Meta’s AI research publications and their Data Policy. No personally identifiable information is intentionally collected.

⚙️ Rate Limiting Policy

While MetaEuro Web Search is not malicious, its sustained crawl rate can overwhelm smaller sites if left unchecked. Therefore, rate-limiting with a threshold of 20–30 requests per second per IP is recommended to prevent resource exhaustion while still allowing legitimate indexing. Meta also advises webmasters to use Crawl-Delay in robots.txt to control pace.

53% of Web Traffic Is Bots in 2026

— Imperva Bad Bot Report 2026

How much of your traffic is automated? Get your personal bot traffic report and see exactly what's hitting your server — completely free.

📊 Get My Bot Report

Sign up in seconds  ·  No card required

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.

✓