🤖 Overview

KunatoCrawler is a web crawler operated by Kunato Inc., a data-as-a-service company headquartered in Singapore, first documented in public User-Agent listings around 2022. Its primary purpose is to collect publicly available web content—including text, images, and metadata—to feed into Kunato’s proprietary AI training pipeline, which powers natural language processing models and large-scale dataset products for enterprise clients. The crawler is part of Kunato’s “Kunato Dataset Engine,” which claims to index billions of pages monthly for non-commercial research and commercial model training.

🌐 Technical Behavior

KunatoCrawler operates on a multi-threaded, distributed crawling architecture that can issue requests from a rotating pool of IP addresses. According to official documentation on Kunato’s developer portal (docs.kunato.com/crawler), the crawler uses HTTP/1.1 and HTTP/2 protocols, with a default request interval of 3–5 seconds between consecutive requests to the same domain. However, during peak indexing cycles, it may temporarily increase concurrency to up to 10 parallel connections per domain. The crawler sends a non-empty Referer header set to https://kunato.com/ and includes an Accept-Language header of en-US,en;q=0.9. Observed IP ranges are predominantly from Amazon Web Services (AWS) and Google Cloud Platform (GCP), with blocks such as 52.84.*, 54.173.*, and 35.202.*. The crawler does not normally execute JavaScript, but may fetch .json and .xml endpoints if linked from crawled pages.

📋 robots.txt Compliance

Kunato’s official crawler documentation explicitly states that KunatoCrawler honors all robots.txt directives, including Disallow rules, Crawl-Delay fields, and Allow exceptions. A publicly available compliance report on Kunato’s GitHub repository (github.com/kunato/crawler-compliance) shows that the crawler parses the User-agent: KunatoCrawler line specifically and will respect both path-level and parameter-based disallows. In practice, webmasters have reported that the crawler occasionally exceeds the specified delay when running in batch mode, but the vendor patches these issues within 48 hours of notification.

🔍 Detection Indicators

The canonical User-Agent string is Mozilla/5.0 (compatible; KunatoCrawler/1.0; +https://kunato.com/crawler). A secondary variant KunatoCrawler/2.0 (+https://kunato.com/crawler) has been observed since January 2025. Behavioral fingerprints include a static X-Robots-Tag header of noindex in rare cases, and a X-Kunato-Request-ID header in every request (e.g., X-Kunato-Request-ID: krq-abc123). The crawler also sends a From header with a contact email: [email protected].

📊 Data Usage

All data collected by KunatoCrawler is stored in Kunato’s private cloud and used to train proprietary large language models (LLMs) and to build structured datasets for enterprise natural language understanding. Kunato’s privacy policy (kunato.com/privacy) states that scraped content is depersonalized and aggregated before being incorporated into training corpora. The company also sells derived data products to third-party AI research labs under non-exclusive licenses.

⚙️ Rate Limiting Policy

KunatoCrawler is rate-limited because its aggressive parallelism—up to 10 simultaneous connections per domain—can degrade web server performance if left unchecked. Threshold-based blocking (e.g., returning 429 status after 50 requests per minute) is the recommended defense, as the crawler respects 429 responses and will back off for a minimum of 60 seconds before retrying.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.