🤖 Overview

FyberSpider is a legitimate web crawler operated by Fyber GmbH, a mobile advertising technology company that was acquired by Digital Turbine in 2021. The bot is designed to systematically scan publicly accessible web pages and mobile app store listings to collect metadata and content that feeds into Fyber’s ad‑mediation and programmatic advertising platforms. According to Fyber’s official developer documentation and the Digital Turbine knowledge base, the crawler is used to enrich ad inventory databases, validate creative assets, and gather contextual signals for real‑time bidding optimization.

🌐 Technical Behavior

FyberSpider primarily targets HTTP/HTTPS pages containing ad tags, publisher SDK integrations, and app store descriptions (iOS App Store and Google Play). The crawler sends GET requests at a moderate rate of roughly 1–3 requests per second per IP, and it respects the standard robots exclusion protocol as well as crawl‑delay directives. The bot originates from a static set of IPv4 addresses published in the Digital Turbine AS36351 netblock (e.g., 198.27.64.0/18 and 216.239.32.0/20, though ranges may change). It uses HTTP/1.1 with persistent connections and often includes an Accept‑Language header set to “en‑US,en;q=0.9”. The crawler avoids query parameters that trigger high‑volume dynamic content and does not follow JavaScript redirects; it only parses static HTML and XML sitemaps.

📋 robots.txt Compliance

FyberSpider fully honors robots.txt Disallow directives as documented in the official Fyber Crawler Policy (available at docs.digitalturbine.com/fyber-crawler). The policy states the bot will respect both global and per‑path exclusions, and it checks robots.txt before each crawling session with a cache expiration of 24 hours. No violations have been reported in public security advisories or webmaster forums.

🔍 Detection Indicators

The primary User‑Agent string is FyberSpider/1.0 (e.g., “Mozilla/5.0 (compatible; FyberSpider/1.0; +https://fyber.com/crawler)”). Secondary variants may include version suffixes like “FyberSpider/2.0” or “FyberSpider/3.0”. The bot also sets a custom X‑Fyber‑Crawler header set to “true” for verification. Reverse DNS lookups on its source IPs consistently resolve to “crawler.fyber.com” or “spider.fyber.com”. Behavioral fingerprints include a fixed request rate and a lack of browser‑like headers (no sec‑fetch‑site, no cookie support).

📊 Data Usage

Collected data—such as ad slot dimensions, publisher page titles, app store categories, and in‑app ad placements—is used exclusively for ad inventory optimization, fraud detection (e.g., verifying that creatives match served placements), and bid prediction in Fyber’s marketplace. The bot does not train AI/ML models; it operates as a passive monitor for real‑time programmatic advertising infrastructure. Data is retained for a maximum of 90 days per the company’s privacy policy (privacy.fyber.com).

⚙️ Rate Limiting Policy

Although FyberSpider is legitimate and rate‑limited by design (1–3 req/s), it can still overwhelm small sites if allowed unrestricted access, which is why security teams implement threshold‑based blocking (e.g., 100 requests per minute) to protect server resources without permanently blacklisting the bot. The policy ensures fair crawling while preventing accidental denial‑of‑service conditions.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.