🤖 Overview

masidani_bot_ is a legitimate web crawler operated by Masidani, a search engine and analytics company based in Japan. First documented in public log files around 2022, its primary purpose is to index web content for Masidani's search engine and provide aggregated analytics data to website owners. The bot is explicitly listed in the robots.txt exclusion standard and is considered a well-behaved agent that respects crawl directives.

🌐 Technical Behavior

The crawler makes HTTP/1.1 requests primarily over IPv4 and IPv6 addresses allocated to Masidani's cloud infrastructure. Observed IP ranges include 103.28.52.0/24 and 2400:8500::/32 (ASN 9729). Requests occur at intervals of 30 to 60 seconds per domain, with bursts during initial discovery of new content. The bot uses HTTPS exclusively and follows redirects, but does not crawl JavaScript-rendered content unless explicitly linked via static HTML. It honors Link rel="nofollow" attributes and avoids URLs containing query strings unless allowed by wildcard rules. The default user-agent string is masidani_bot_/1.0 with variations like masidani_bot_/2.0 for updated crawlers.

📋 robots.txt Compliance

Based on Masidani's official documentation (available at masidani.com/crawler-policy), the bot fully respects Disallow and Allow directives in robots.txt. It also adheres to Crawl-delay instructions with a minimum of 10 seconds. Since 2023, the bot reads the X-Robots-Tag HTTP header and supports meta name="robots" content directives for page-level control.

🔍 Detection Indicators

Primary User-Agent string: Mozilla/5.0 (compatible; masidani_bot_/1.0; +https://masidani.com/bot). Alternative strings include masidani_bot_/2.0 and masidani-bot for older deployments. The bot does not set a via header but includes From: [email protected] in request headers. It also appends a X-Masidani-Bot: true header for verification. Reverse DNS lookups resolve to *.masidani.cloud subdomains.

📊 Data Usage

Collected data is used exclusively for search engine indexing and Masidani Analytics, a free service that provides traffic insights to webmasters. According to Masidani's privacy policy (masidani.com/privacy), content is cached for up to 30 days and is not used for AI training or sold to third parties. The bot also contributes to the Masidani Link Database for web graph analysis.

⚙️ Rate Limiting Policy

Despite its legitimate nature, masidani_bot_ is rate-limited because its regular crawling can spike to 100 requests per minute on large sites during initial indexing. Implementing threshold-based blocking ensures fair resource distribution and prevents accidental load spikes, while still allowing the bot to index content over longer periods.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.