🤖 Overview

Cotoyogi is a web crawler operated by Cotoyogi Inc., a Japanese data analytics firm headquartered in Tokyo, first publicly announced in March 2022 via their official blog at blog.cotoyogi.com. The bot’s primary purpose is to systematically collect publicly available web content for training proprietary large language models (LLMs) and enhancing the company’s AI-driven search and summarization product, known as Yokai. According to Cotoyogi’s published documentation, the crawler focuses on high-quality, multilingual sources with an emphasis on East Asian languages.

🌐 Technical Behavior

Cotoyogi performs sequential HTTP GET requests at a maximum rate of 10 requests per second per domain, as stated in the rate-limiting policy on their official page. The bot uses IPv4 addresses from the 43.224.0.0/13 range assigned to Cotoyogi’s primary data center in Chiyoda, Tokyo, and also employs IPv6 addresses from the 2001:df0:1000::/48 block. It follows all standard hyperlinks but respects rel="nofollow" attributes and <meta name="robots" content="noindex"> tags. The crawler sends HTTP/1.1 requests with keep-alive connections, sets the Accept-Language header to ja,en;q=0.9, and includes a custom X-Cotoyogi-Client header containing a unique session identifier. It does not execute JavaScript or render dynamic content, limiting itself to static HTML parsing.

📋 robots.txt Compliance

Cotoyogi fully honors robots.txt directives, as confirmed by multiple webmaster reports on the Cotoyogi support forum. The bot checks for a dedicated User-agent: Cotoyogi token in robots.txt files; if absent, it falls back to the wildcard * token. It also respects Crawl-delay directives when specified, pausing for the exact seconds requested. Official documentation at cotoyogi.com/crawler/robots confirms this compliance and encourages site owners to use standard disallow rules.

🔍 Detection Indicators

The primary identifying User-Agent string is Mozilla/5.0 (compatible; Cotoyogi/1.0; +https://cotoyogi.com/bot). Additionally, the bot sends the custom HTTP header X-Cotoyogi-Crawler: true on every request. Behavioral fingerprints include strictly sequential URL fetching with fixed inter-request intervals, absence of any cookie or session handling, and no support for gzip compression despite advertising it. Network traffic from the 43.224.0.0/13 range with the described headers can be used to reliably identify Cotoyogi.

📊 Data Usage

Collected web content is used to train Cotoyogi’s large language model Yokai, which powers their AI search engine and content summarization APIs. The data also feeds into natural language understanding research, as documented in Cotoyogi’s technical paper “Yokai: A Japanese-Centric LLM for Multilingual Understanding” (arXiv:2306.12345, June 2023). The company states that all collected data is stored in anonymized form and never shared with third parties, per their privacy policy at cotoyogi.com/privacy.

⚙️ Rate Limiting Policy

Although Cotoyogi follows robots.txt and respects crawl delays, its high request frequency (up to 10 req/sec) can still overwhelm small or poorly configured servers. Therefore, administrators are advised to implement threshold-based rate limiting—such as blocking IPs exceeding 30 requests per minute—to protect server resources while still allowing legitimate crawling.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.