🤖 Overview

nicerspro is a web crawler operated by NICER Inc., a Japanese data analytics and search technology company based in Tokyo, first publicly documented in 2015. Its purpose is to collect publicly accessible web content for NICER’s proprietary search engine index and to feed their AI-driven market research platform, NICER Insights, which provides trend analysis and competitor intelligence to corporate subscribers.

🌐 Technical Behavior

nicerspro runs on a distributed cluster of servers in NICER’s Tokyo and Osaka data centers, using IPv4 addresses from ASN 131921 (NICER Inc.) and IPv6 from the 2400:cb00::/32 prefix. The crawler employs a custom HTTP client based on Python’s requests library, supporting HTTP/1.1 and HTTP/2 protocols. It requests pages at a moderate rate of 5–10 requests per second per IP, with a maximum of 50 concurrent connections, and respects HTTP caching directives such as Cache-Control and Last-Modified. The bot typically initiates crawls during Japanese business hours (JST, UTC+9) and reduces activity on weekends, indicating a human-monitored operation. Official documentation from NICER’s developer portal at developer.nicer.com confirms the crawler obeys the Host header for virtual hosting and does not follow JavaScript or dynamically loaded content.

📋 robots.txt Compliance

According to NICER’s published crawler policy at nicer.com/crawler-policy, nicerspro fully respects both Disallow and Allow directives in robots.txt. It also honors the Crawl-delay directive, applying a default delay of two seconds when none is specified, and re-fetches robots.txt every 24 hours during active crawling cycles.

🔍 Detection Indicators

The primary User-Agent string is nicerspro/1.0 (case-sensitive). Additional identifying headers include a non-standard X-Nicer-Client header containing a numerical identifier, and a valid From header set to [email protected]. Traffic analysis shows the bot consistently uses a small pool of around 30 IPv4 addresses, all resolving to n.nicer.com PTR records.

📊 Data Usage

Collected data populates NICER’s search engine at search.nicer.com and trains machine learning models for NICER Insights. NICER’s privacy policy at nicer.com/privacy states that any personally identifiable information discovered during crawling is automatically redacted, and data is retained for a maximum of 90 days for index freshness.

⚙️ Rate Limiting Policy

Because nicerspro can sustain 5–10 requests per second per IP and occasionally burst during peak hours, web administrators should implement rate limiting at 30 requests per second per IP. Threshold-based blocking is recommended to prevent resource exhaustion on shared hosting while still allowing legitimate data collection.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.