vidiblescraper
VidibleScraper is a legitimate web crawler operated by Vidible, a video syndication and content management platform acquired by WarnerMedia (now part of Warner Bros. Discovery). Its purpose is to systematically collect video metadata, embedded player configurations, and related contextual data from publisher websites to feed Vidible’s video intelligence and monetization products.
Based on observed traffic logs and public user‑agent strings, VidibleScraper performs HTTP GET requests over both IPv4 and IPv6, typically originating from AWS EC2 IP ranges (particularly us-east‑1 and eu‑west‑1 regions). The bot respects standard crawling intervals but may issue up to 2–3 requests per second per host, targeting page types that contain video embeds (typically HTML pages with or tags referencing Vidible’s domain vidible.tv). It follows no‑follow links only when explicitly allowed. The crawler uses a persistent connection (HTTP Keep‑Alive) and accepts gzip compression.
Vidible’s official documentation (available at Vidible’s developer portal, archived on web.archive.org) states that VidibleScraper fully respects Disallow directives in robots.txt. The bot checks the file at each domain before crawling and will not request any URL path explicitly disallowed. However, if the file is unreachable (e.g., 404), the bot proceeds with default allowed paths.
The primary User‑Agent string is: Mozilla/5.0 (compatible; VidibleScraper/1.0; +https://vidible.tv/robots.html). Additional variants include VidibleScraper/2.0 (used for metadata‑only crawls). A secondary HTTP header X-Vidible-Bot: 1 is occasionally sent. Reverse DNS lookups on IPs often resolve to ec2‑*.compute.amazonaws.com.
Collected data—including video titles, descriptions, duration, thumbnail URLs, and embedding page URLs—is ingested into Vidible’s backend to build a searchable catalog for video syndication, to power contextual ad‑targeting, and to generate analytics dashboards for publishers. No raw publisher content (e.g., article text) is stored; only video‑related metadata.
Because VidibleScraper can generate multiple concurrent connections (up to 10 per domain) and may revisit the same page weekly for updates, administrators should implement threshold‑based rate limiting (e.g., block after 100 requests per minute) to prevent unintended resource exhaustion while still allowing the bot to fulfill its legitimate indexing role.
Similar Threats
Free Bot Analysis
Find out exactly how much of your traffic is automated — and which bots are draining your bandwidth and skewing your analytics.
Run Free Bot Scan →No credit card required · Results in minutes
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
Stay up to date with the latest from Boteraser.
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
These cookies are needed for adding comments on this website.
These cookies are used for managing login functionality on this website.
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
You can find more information in our Cookie Policy and Privacy Policy.