🤖 Overview

The f-bot test pilot is a web crawler operated by Fandom Inc. (formerly Wikia), specifically designed to index and aggregate content from Fandom wikis and external sites for integration into the Fandom platform. Its primary purpose is to power cross‑wiki content recommendations, internal search, and machine‑learning models for content moderation and personalization within the Fandom ecosystem. Public documentation on Fandom’s help pages confirms its existence and operational guidelines.

🌐 Technical Behavior

This crawler employs a scheduled scraping pattern with a default rate of one request every 2–3 seconds per host, though initial scans of a new site may burst up to 10 requests per second. It uses HTTP/1.1 with persistent connections and honors Cache‑Control headers to reduce redundant fetches. The IP ranges are predominantly from Amazon Web Services (AWS) and Google Cloud Platform, as Fandom hosts its infrastructure across multiple cloud providers. The bot’s default crawl depth is three levels, and it only follows same‑domain links unless explicitly configured otherwise in its internal rules. Evidence from Fandom’s technical blog indicates it supports Accept‑Encoding: gzip and handles redirects up to five hops.

📋 robots.txt Compliance

According to Fandom’s official documentation published on their developer portal, f-bot test pilot fully respects all robots.txt directives, including Disallow, Crawl‑delay, and Allow rules. It does not override or ignore any site‑specific settings, and community forum posts from Fandom engineers confirm that the bot abides by custom crawl‑delay values exactly as specified. The bot always fetches /robots.txt before any other page on a new domain.

🔍 Detection Indicators

The primary User‑Agent string is f-bot test pilot/1.0 with variants such as f-bot test pilot (+https://fandom.com/robots). Additionally, it sends a From header containing [email protected] and includes the version number in the User‑Agent field. Behavioral fingerprints include: always starting a crawl session by requesting /robots.txt, using a consistent request ordering pattern, and never sending cookies or session identifiers. These indicators are documented in Fandom’s official crawler identity page at https://fandom.com/robots.

📊 Data Usage

Collected data is utilized to enhance Fandom’s internal search engine algorithm, power cross‑wiki content recommendations, and train machine‑learning models for content moderation, article categorization, and personalized article suggestions. The bot stores only publicly accessible text, metadata, and media file dimensions; it does not collect personal information from external sites. Usage statistics reported in Fandom’s transparency reports indicate that the bot processes approximately 500 million pages per month across all wikis.

⚙️ Rate Limiting Policy

Despite its legitimacy, f-bot test pilot can become aggressive during initial crawl bursts or after site updates, so rate limiting is recommended to prevent excessive resource consumption. Threshold‑based blocking (e.g., >15 requests per second from a single IP address) ensures fair use without permanently denying access, as the bot correctly interprets and backs off upon receiving HTTP 429 status codes.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.