🤖 Overview

The webspider is a legitimate web crawler operated by Web.com (now a subsidiary of Newfold Digital), primarily used for SEO analysis, site monitoring, and security scanning services offered to Web.com’s hosting and domain customers. This bot systematically indexes publicly accessible pages to generate performance audits, detect broken links, and identify security vulnerabilities on behalf of subscribing website owners. It is explicitly documented in Web.com’s official support resources as a non‑malicious agent designed to improve website health.

🌐 Technical Behavior

Webspider sends HTTP/1.1 GET requests at a variable rate that can reach several requests per second, depending on the target site’s response times and the depth of the crawl. The crawler follows both a tags and href links, respects noindex meta tags, and identifies its origin IP ranges from Web.com’s AS‑number (e.g., AS 14955 for Newfold Digital). Official documentation lists IP blocks such as 64.70.19.0/24 and 208.91.199.0/24, though ranges may shift. It uses TLS 1.2+ for HTTPS sites and defaults to port 80/443. A typical crawl session lasts 10–30 minutes and may request a sitemap before starting.

📋 robots.txt Compliance

Webspider honors standard robots.txt Disallow directives by default, as confirmed by Web.com’s public crawl policy page and independent tests. Website operators can also use Crawl‑Delay directives to throttle the bot’s request rate, and it acknowledges Allow rules for specific paths. The company advises that ignoring robots.txt can lead to excessive load, but the bot is designed to comply.

🔍 Detection Indicators

The primary User‑Agent string is Web.com Spider/1.0 or variations like webspider/1.0 or WebcomSpider/1.0. Behavioral fingerprints include a consistent request pattern of fetching the homepage, then URLs found on the page, with a typical 1–3 second delay between requests. The HTTP header X‑Forwarded‑For may be absent, but the Connection header is often set to keep‑alive. Reverse DNS lookups on its source IPs usually resolve to spider.web.com or similar subdomains.

📊 Data Usage

Collected data—including page titles, meta descriptions, response status codes, and security headers—is fed into Web.com’s Site Analyzer product and used to generate monthly reports for customers. The information is not sold externally; it solely improves the user’s own website optimization, helps detect malware, and validates HTTPS configuration. Web.com stores crawl results for up to 90 days and aggregates anonymized metrics for internal service improvement.

⚙️ Rate Limiting Policy

Because webspider can issue requests in bursts and sometimes ignores Crawl‑Delay if not set, it is rate‑limited to prevent server resource exhaustion. Administrators are advised to implement threshold‑based blocking (e.g., more than 10 requests per second) while still allowing the bot through when it respects robots.txt, ensuring service continuity for legitimate analysis.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.