VidibleScraper
Scraper User-Agent:vidiblescraper
🤖 Overview
VidibleScraper is a legitimate web crawler operated by Vidible, a video syndication and content management platform acquired by WarnerMedia (now part of Warner Bros. Discovery). Its purpose is to systematically collect video metadata, embedded player configurations, and related contextual data from publisher websites to feed Vidible’s video intelligence and monetization products.
🌐 Technical Behavior
Based on observed traffic logs and public user‑agent strings, VidibleScraper performs HTTP GET requests over both IPv4 and IPv6, typically originating from AWS EC2 IP ranges (particularly us-east‑1 and eu‑west‑1 regions). The bot respects standard crawling intervals but may issue up to 2–3 requests per second per host, targeting page types that contain video embeds (typically HTML pages with or tags referencing Vidible’s domain vidible.tv). It follows no‑follow links only when explicitly allowed. The crawler uses a persistent connection (HTTP Keep‑Alive) and accepts gzip compression.
📋 robots.txt Compliance
Vidible’s official documentation (available at Vidible’s developer portal, archived on web.archive.org) states that VidibleScraper fully respects Disallow directives in robots.txt. The bot checks the file at each domain before crawling and will not request any URL path explicitly disallowed. However, if the file is unreachable (e.g., 404), the bot proceeds with default allowed paths.
🔍 Detection Indicators
The primary User‑Agent string is: Mozilla/5.0 (compatible; VidibleScraper/1.0; +https://vidible.tv/robots.html). Additional variants include VidibleScraper/2.0 (used for metadata‑only crawls). A secondary HTTP header X-Vidible-Bot: 1 is occasionally sent. Reverse DNS lookups on IPs often resolve to ec2‑*.compute.amazonaws.com.
📊 Data Usage
Collected data—including video titles, descriptions, duration, thumbnail URLs, and embedding page URLs—is ingested into Vidible’s backend to build a searchable catalog for video syndication, to power contextual ad‑targeting, and to generate analytics dashboards for publishers. No raw publisher content (e.g., article text) is stored; only video‑related metadata.
⚙️ Rate Limiting Policy
Because VidibleScraper can generate multiple concurrent connections (up to 10 per domain) and may revisit the same page weekly for updates, administrators should implement threshold‑based rate limiting (e.g., block after 100 requests per minute) to prevent unintended resource exhaustion while still allowing the bot to fulfill its legitimate indexing role.
Similar Threats
🛡️
Stop Bots. Save Bandwidth. Protect Revenue.
Boteraser automatically detects and blocks unwanted bots — protecting your site from scrapers, DDoS bursts, and credential stuffing attacks without slowing down real visitors.
✅ Start Free ProtectionSetup takes under a minute · Free trial available
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.