🤖 Overview

The ichiro crawler is operated by ICHINO Search Co., Ltd. (formerly a division of Goo/NTT Resonant) for the purpose of building and updating the search index of the Japanese search engine Ichiro (ichiro.jp). Announced in documentation as early as 2006, this bot is designed to discover and index publicly accessible web pages, specifically optimized for Japanese-language content, and functions as a legitimate, non-malicious agent in the web ecosystem.

🌐 Technical Behavior

ichiro employs standard HTTP GET requests with support for both HTTP/1.1 and HTTP/2 protocols. Crawl patterns include a default interval of approximately 1–2 seconds per request, though it can increase frequency during initial site discovery or re-crawl cycles. The bot uses IP addresses drawn from ranges registered to Japanese data center providers such as IIJ, KDDI, and NTT Communications; these ranges are publicly documented in WHOIS records and are subject to change. It advertises a Via header of 1.1 ichiro in some requests and always includes a User-Agent string. The crawler follows HTTP redirects up to 5 hops and handles robots.txt directives before each crawl session.

📋 robots.txt Compliance

According to ICHINO Search technical documentation published at https://www.ichiro.jp/crawler/, the bot fully honors Disallow directives in robots.txt. It also respects Crawl-delay directives when present, and will pause between requests accordingly. Evidence from multiple site logs confirms that ichiro does not crawl paths listed in the Disallow: rule, though it is known to re-fetch robots.txt every 24 hours to check for updates.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; ichiro/2.0; +http://www.ichiro.jp/crawler_info.html) for version 2.0, and ichiro/1.0 for older instances. Behavioral fingerprints include a consistent header order (User-Agent, Accept, Accept-Language, Accept-Encoding) and the absence of client-side JavaScript execution. The bot also sends a From field in some requests, set to [email protected], as documented in their official crawler information page.

📊 Data Usage

All collected data—including page content, metadata, and links—is used exclusively to populate the Ichiro Search index and to provide search results for Japanese-language queries. The data is not repurposed for AI training, third-party analytics, or any advertising model according to ICHINO Search’s privacy policy. The company retains indexed content temporarily for refresh cycles and purges outdated pages within 30 days of a 410 Gone response.

⚙️ Rate Limiting Policy

ichiro is rate-limited because its aggressive re-crawl schedules can consume significant server resources, especially on small to medium websites; threshold-based blocking at 10 requests per second from a single IP is a common administrative policy to ensure fair bandwidth allocation without permanently denying access to a legitimate search engine bot.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.