🤖 Overview

fu-nbi is a web crawler operated by Meta Platforms, Inc. (formerly Facebook) as part of its News Bot Index (NBI) system, publicly documented at https://developers.facebook.com/docs/sharing/bot/. Its explicit purpose is to index news articles and publicly available content for the Facebook News feature, enabling link previews and content discovery within the platform.

🌐 Technical Behavior

The crawler sends HTTP GET requests using User-Agent strings like "nbi", "fu-nbi", "facebookexternalhit/1.1" and "Facebot". It originates from Meta's published IP ranges, including blocks such as 69.171.224.0/19, 93.158.128.0/17, and 31.13.24.0/21 (see https://developers.facebook.com/docs/sharing/bot/ip-addresses). Requests target Open Graph, JSON-LD, and standard HTML meta tags to extract article headline, description, image, and author. The crawler does not execute JavaScript and typically makes one request per URL with a polite delay between consecutive requests, often during low-traffic periods. According to Meta's documentation, it may follow redirects and accept gzip encoding.

📋 robots.txt Compliance

Meta states that fu-nbi fully respects robots.txt directives, using the User-Agent token "nbi" or "facebookexternalhit". To block it, webmasters should add User-agent: nbi and Disallow: / in robots.txt. There is no documented exception for content necessary for indexing, though Facebook reserves the right to crawl public data for legitimate purposes.

🔍 Detection Indicators

Key User-Agent strings: "nbi", "fu-nbi", "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)", and "Facebot". Reverse DNS lookups reveal hostnames like "nbi.facebook.com". IP addresses belong to AS32934 and AS54113. Behavioral fingerprint: rapid bursts of 3–5 requests from a single IP, followed by a pause, often requesting many pages on the same domain within minutes.

📊 Data Usage

Extracted metadata is used to generate link previews in Facebook posts, populate the Facebook News feed, and improve algorithmic content recommendations. Meta also uses this data for training AI models related to news summarization, topic classification, and understanding content relevance. Data is stored temporarily (typically up to 30 days) and may be aggregated for analytics.

⚙️ Rate Limiting Policy

Rate limiting of fu-nbi is recommended because its crawl volume can spike during major news events, potentially overwhelming origin servers. A threshold-based policy (e.g., 10–15 requests per second per IP with a burst limit of 20) ensures fair resource allocation without blocking legitimate indexing, as documented in best practices from OWASP and CDN providers.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.