🤖 Overview

Kuloko-Bot is a web crawler operated by Kuloko Inc., a Japanese search-engine company based in Tokyo, used to index web content for the Kuloko search engine (kuloko.jp). First documented publicly in 2005, it focuses on Japanese-language and multilingual pages to power keyword-based search results, competing with services like Yahoo! Japan and Google in the Japanese market.

🌐 Technical Behavior

Kuloko-Bot crawls by following links from seed lists and sitemaps, using an asynchronous multi-threaded architecture that typically issues requests at intervals of 1-5 seconds per domain, though aggregate crawl rates can exceed 50 requests per second across multiple hosts. According to official documentation, it employs a variant of the Depth-First Search algorithm and respects Crawl-delay directives. IP addresses originate primarily from the ASN AS2518 (BIGLOBE) and ASN AS7506 (INTER-MESH), covering ranges such as 210.150.0.0/16 and 203.138.0.0/16, but Kuloko also leases IPs from other Japanese ISPs. Connections use HTTP/1.1 and HTTPS, with a default Accept-Language header of ja to indicate Japanese preference.

📋 robots.txt Compliance

Kuloko-Bot fully honors robots.txt directives, as verified by community tests and Kuloko's own documentation. It checks the file each time it begins crawling a domain and caches it for the duration of the crawl session. No reports of ignoring Disallow rules have been found; the bot abides by standard allowances and disallowances.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Kuloko-Bot/1.0; +https://www.kuloko.jp/bot.html), though some variants omit the Mozilla prefix. Secondary identifiers include the X-Robot-Id header set to KulokoBot and a consistent IP range as mentioned above. Behavioral fingerprinting reveals a short inter-request delay and a preference for HTML content over JavaScript-heavy pages.

📊 Data Usage

Collected data is used exclusively for indexing and ranking web pages in the Kuloko search engine, which serves both general web search and a specialized Japanese-language vertical. The indexed content is not used for AI model training, advertising profiling, or sold to third parties, according to Kuloko's privacy policy published at kuloko.jp/privacy. Metadata such as page titles and meta descriptions are stored in the search index for query matching.

⚙️ Rate Limiting Policy

Rate-limiting of Kuloko-Bot is recommended only when its crawl load threatens site performance, as the bot adheres to rate limits set via Crawl-delay. For sites without a robots.txt directive, thresholds of 5 requests per second per IP are a common safeguard against excessive traffic from the bot's multi‑threaded crawler.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.