🤖 Overview

wbdbot is a legitimate web crawler operated by Webedia, a French digital media group that owns properties such as Jeuxvideo.com, AlloCiné, and PureBreak. Its primary purpose is to index publicly accessible web content to feed Webedia’s internal recommendation engine, search functionality, and content aggregation systems across their network of websites. The bot was first documented publicly around 2016 and is explicitly managed by Webedia’s technical team to support their product ecosystem, not for external AI training.

🌐 Technical Behavior

wbdbot performs standard HTTP/1.1 GET requests with a consistent crawl delay of approximately 2 seconds between requests, as verified in official documentation on wbdbot.com. It operates from IP ranges registered under Webedia’s autonomous system (AS197422), with typical outbound IPs originating from France. The crawler follows standard robots.txt directives and supports the Crawl-Delay directive. It does not execute JavaScript or render pages, focusing solely on static HTML and linked resources. Webedia’s crawl policy states a maximum of 10 requests per second per IP, though in practice the bot is observed to stay well below that threshold.

📋 robots.txt Compliance

Webedia officially states on its bot information page that wbdbot fully complies with the Robots Exclusion Standard. The crawler is programmed to honor all Disallow directives and respects any Crawl-Delay specified in robots.txt. Multiple site owner reports confirm that wbdbot does not attempt to access blocked paths once a valid robots.txt is served.

🔍 Detection Indicators

The primary User-Agent string is “wbdbot/1.0 (compatible; +https://www.wbdbot.com/bot)”. Additional identifying headers include a From header containing the email address [email protected]. Behavioral fingerprints include uniform request intervals, no referrer spoofing, and a consistent lack of cookies. Log analysis shows wbdbot rarely, if ever, requests binary files or dynamic endpoints like /api/. The bot’s IP ranges are registered under the organization name “Webedia” in WHOIS records.

📊 Data Usage

Data collected by wbdbot is used exclusively for internal product improvement within Webedia’s ecosystem: improving search relevance, refining content recommendations on sites like Jeuxvideo.com, and powering aggregate editorial tools. The data is not used for training external large language models or sold to third parties. Webedia’s privacy policy confirms that raw crawl data is retained for a maximum of 90 days and is not shared beyond its corporate group.

⚙️ Rate Limiting Policy

wbdbot is rate-limited by default to protect web servers from excessive load, with Webedia recommending a threshold of 100 requests per minute per IP before implementing temporary blocking. This policy ensures fair resource usage while allowing the bot to fulfill its legitimate indexing role.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.