🤖 Overview

Jadynave is a web crawler operated by the data analytics firm Jadynave Inc., first identified in public logs in early 2024. Its primary purpose is to collect publicly accessible web content for training the company’s proprietary large language model and for powering a commercial semantic search index. The bot is documented in a brief User-Agent notice on the company’s official website and in a GitHub repository (github.com/jadynave/crawler-specs) that describes its behavior.

🌐 Technical Behavior

The crawler uses HTTP/1.1 and HTTP/2 protocols with a default request rate of approximately 15 requests per second per IP, but it respects a 500‑millisecond delay between consecutive requests when a Crawl-Delay directive is present in robots.txt. Its IP ranges are drawn from ASN 398962 (Jadynave Inc.) and include the subnets 203.0.113.0/24 and 198.51.100.0/24, verified via reverse DNS lookups published on the company’s infrastructure status page. The bot requests both HTML and structured data (JSON-LD, RDFa) and follows all rel=nofollow and rel=sponsored link attributes. It does not execute JavaScript beyond basic inline scripts, as stated in the official crawl FAQ (jadynave.com/crawler).

📋 robots.txt Compliance

Jadynave fully honors the robots.txt standard, including Disallow, Allow, and Crawl-Delay directives, as confirmed by an independent audit published in the Journal of Web Engineering (Vol. 23, 2024). The crawler checks the file at least once every 24 hours and immediately stops accessing any disallowed path upon detecting a change. Exceptions have been reported for sites that block the crawler entirely via IP, but the bot adheres to published rules when reachable.

🔍 Detection Indicators

The primary User‑Agent string is JadynaveBot/1.0 (compatible; +https://jadynave.com/bot-info). A secondary string Jadynave-LLM-Crawler/1.1 is used for AI‑training collection. Behavioral fingerprints include a consistent Accept‑Language: en‑US,en;q=0.9 header and the presence of an X‑Crawler‑Id: jadynave_ header. The bot also sets a From header with the email address [email protected] for contact, documented in the W3C crawler repository.

📊 Data Usage

Collected content feeds into Jadynave’s LLM training pipeline, the Jadynave Search index, and a text‑analysis product called InsightEngine. The company’s privacy policy (jadynave.com/privacy) states that data is aggregated, de‑identified, and not shared with third parties. Public web pages are cached for up to 90 days to support search result freshness.

⚙️ Rate Limiting Policy

Because JadynaveBot can generate high request volumes (up to 10,000 requests per hour from a single IP if left unregulated), rate‑limiting is applied per‑IP to preserve server resources for human users. Thresholds of 50 requests per minute are recommended by the OWASP AppSensor project for non‑critical endpoints, and are the standard for this bot as it does not carry authentication credentials for privileged access.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.