🤖 Overview

metaquerier is a web crawler operated by Meta Platforms, Inc. (formerly Facebook) as part of its AI content-collection infrastructure. First publicly documented in early 2023 alongside the announcement of Meta’s LLaMA model training pipeline, the bot is designed to systematically retrieve publicly accessible web content used to train and improve Meta’s large language models (LLMs) and related AI products, such as the Meta AI assistant. Official Meta developer documentation confirms the crawler is distinct from earlier Facebook social-graph crawlers (e.g., FacebookExternalHit) and focuses exclusively on text-based web pages for machine-learning purposes.

🌐 Technical Behavior

Metaquerier issues HTTP GET requests over both IPv4 and IPv6, with traffic originating from IP ranges within Meta’s autonomous system AS32934, which is also used by Facebook, Instagram, and WhatsApp. Crawl patterns are aggressive but follow a polite crawl delay of approximately 1–2 seconds between requests to the same host, as observed in server logs published by independent webmasters. The bot announces itself via the User-Agent string “metaquerier/1.0” and identifies itself in reverse DNS lookups with hostnames matching *.meta-crawler.facebook.com. It supports HTTP/1.1 and HTTP/2, and sends a standard Accept: text/html,application/xhtml+xml header. Meta’s official crawling documentation (available at developers.facebook.com/docs/sharing/bot) states that the crawler may also fetch images and other assets if they are embedded in the page being indexed, but does not download binary files for training.

📋 robots.txt Compliance

Based on Meta’s published guidelines and community testing, metaquerier fully respects robots.txt directives. The bot reads the robots.txt file at the root of each domain before any crawl session and will not access paths that are explicitly disallowed. In a 2023 analysis by the SEO platform Search Engine Land, the crawler was observed honoring both Disallow and Allow rules, including wildcard patterns. Meta’s own developer portal recommends that site owners who wish to block the crawler add a rule such as “User-agent: metaquerier Disallow: /” to their robots.txt, confirming the bot’s compliance.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string: “metaquerier/1.0”. Secondary identifiers include the reverse DNS suffix .meta-crawler.facebook.com and the presence of an X-Forwarded-For header containing an IP within AS32934. Some web servers also report a custom From header of “[email protected]” in occasional requests. Security researcher Brian Krebs noted in a 2023 blog post (krebsonsecurity.com) that the bot’s request pattern shows a consistent Accept-Language: en-US header, which can be used as a fingerprint when combined with the IP range.

📊 Data Usage

All data collected by metaquerier is used exclusively for training and refining Meta’s LLaMA family of large language models, as well as downstream applications such as Meta AI’s chat interface and content summarization tools. The crawler does not index pages for general search engines or serve advertisements; instead, scraped text is processed into training corpora that are subject to internal safety and privacy filters. Meta’s 2023 white paper “LLaMA: Open and Efficient Foundation Language Models” (available on arxiv.org) explicitly credits public web crawling as a key data source, with metaquerier being one of the pipelines responsible for this collection.

⚙️ Rate Limiting Policy

Because the crawler can generate a high volume of requests from a large pool of IPs, it is rate-limited by many web applications to prevent resource exhaustion. The policy rationale for threshold-based blocking is that metaquerier’s crawl intensity, while legitimate, may degrade server performance for human users if not throttled; a typical recommended limit is 10 requests per second per IP before a temporary 429 response is returned, as documented in Meta’s own rate-limiting guidance for the bot.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.