🤖 Overview

cybernavi_webget is a web crawler operated by CyberNavi Inc., a Japanese digital marketing and SEO analytics company headquartered in Tokyo. Its primary purpose is to systematically fetch web pages to populate the CyberNavi backlink database, site audit reports, and competitive intelligence dashboards offered through their commercial platform. The bot is documented on the official CyberNavi website as part of their “Crawler” service, with a dedicated page describing its behavior and contact information for site owners.

🌐 Technical Behavior

The crawler initiates requests using HTTP/1.1 with a default User-Agent string of cybernavi_webget (case‑sensitive). It sends a Crawl‑Delay header value of 10 seconds when defined in robots.txt, but otherwise respects a default delay of 5 seconds between consecutive requests to the same host. The bot resolves from IP ranges belonging to AS2516 (KDDI Corporation) in Japan, with additional blocks from AS9607 (BIGLOBE Inc.), which are publicly listed in the Japanese Network Information Center (JPNIC) database. It fetches pages at a maximum rate of 60 requests per minute per domain, and deliberately avoids crawling images, CSS, or JavaScript files unless explicitly linked. The user‑agent string does not include a version number, but the crawler appends a trailing comment with contact email: ([email protected]).

📋 robots.txt Compliance

According to the official CyberNavi documentation, cybernavi_webget strictly honors Disallow directives found in robots.txt files. It also respects the Crawl‑Delay directive, but only if the value is set to 10 seconds or less; any higher value is treated as the maximum (10 seconds). The crawler does not attempt to bypass restrictions and will immediately cease crawling a resource upon encountering a Disallow rule in the site’s root robots.txt.

🔍 Detection Indicators

The definitive detection indicator is the exact User‑Agent string: cybernavi_webget (without a slash or version suffix). Log entries will also show the referrer header set to https://www.cybernavi.co.jp/ for most requests. The crawler’s IP addresses consistently resolve to reverse DNS names ending in .kddi.ne.jp or .biglobe.ne.jp. Additionally, the HTTP Accept header is always text/html,application/xhtml+xml, and the bot does not support gzip encoding.

📊 Data Usage

All harvested content is ingested into the CyberNavi Search Cloud platform, where it is indexed for three distinct products: Backlink Checker (identifying referring domains), Site Audit (evaluating page structure and broken links), and Keyword Rank Tracker (monitoring search engine positions). The data is aggregated and stored for a maximum of 90 days, after which raw page text is discarded; only metadata (URLs, titles, timestamps) is retained indefinitely. No personally identifiable information (PII) is collected, and the company states that crawled content is not used for AI training or machine learning model development.

⚙️ Rate Limiting Policy

Rate Limiting is recommended for cybernavi_webget primarily because the bot’s default crawl rate of 60 requests per minute can overload smaller sites, particularly those with low‑bandwidth hosting. The policy rationale is to apply a threshold‑based block (e.g., return 429 Too Many Requests) only when the crawler exceeds 100 requests per minute, which indicates a misconfiguration or an anomalous burst – a fair balance between preserving site performance and allowing legitimate SEO data collection.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.