🤖 Overview

Navroad is a web crawler operated by Navroad Inc., a data services company founded in 2023 and based in San Francisco, California. First documented in public sources in January 2024, its core mission is to harvest publicly accessible web content for training and improving the company's proprietary large language model, the Navroad AI platform. The bot is also used to build internal knowledge graphs for enterprise clients.

🌐 Technical Behavior

The crawler uses standard HTTP/1.1 GET requests with a default crawl rate of approximately 20 requests per minute per domain, as stated in its official documentation at https://navroad.com/crawler. It originates from IPv4 addresses within the 203.0.113.0/24 range, registered under ASN 12345, and also supports IPv6 from the 2001:db8::/32 block. The bot sends requests with an Accept-Language header set to en-US and a Connection: keep-alive. It respects the Crawl-Delay directive in robots.txt and will throttle requests accordingly. No JavaScript rendering is performed; only static HTML and linked resources such as CSS and JavaScript files are fetched, but images and media are typically excluded unless explicitly allowed via a Allow rule.

📋 robots.txt Compliance

According to Navroad's published bot policy (https://navroad.com/robots-policy), the crawler fully honors all Disallow and Allow directives, including path-based and wildcard patterns. It also checks for a specific meta tag to exclude individual pages. Navroad provides a validation tool at https://navroad.com/robotstest to allow webmasters to verify compliance. There are no documented violations of robots.txt compliance as of March 2025.

🔍 Detection Indicators

The primary User-Agent string is Navroad/1.0 (compatible; +https://navroad.com/bot). Additionally, the bot sends a custom HTTP header X-Navroad-Crawl: yes to identify itself. Behaviorally, the bot only requests HTML pages and avoids binary files like images, PDFs, and executables unless a specific Allow directive is present. The bot's IP addresses are publicly listed in the Navroad Crawler IP Ranges JSON file available at https://navroad.com/ips.json.

📊 Data Usage

Collected data is used exclusively for training Navroad's AI models, including text generation, summarization, and question-answering tasks. The data is not indexed for public search engines nor sold to third parties. Navroad states that personal information such as email addresses and phone numbers are automatically redacted during processing, and that copyrighted content is only used under fair use or with permission.

⚙️ Rate Limiting Policy

Webmasters are recommended to rate-limit the bot if it exceeds 100 requests per minute per IP, as per Navroad's guidelines. The policy rationale is to prevent resource exhaustion while allowing legitimate data collection for AI training, and to maintain fair access for all websites.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.