🤖 Overview

The e-societyrobot is a web crawler operated by e-society, Inc., a Japanese company specializing in large-scale language model training and AI data services. Its primary purpose is to systematically collect publicly available web content for training and improving the company’s proprietary generative AI models and natural language processing systems. The bot feeds data into the e-society AI platform, which is used for enterprise search and content summarization products.

🌐 Technical Behavior

According to the official documentation published at https://www.e-society.co.jp/robot/, the e-societyrobot issues requests at an average rate of 10–20 requests per second from IP addresses primarily in the 103.37.64.0/22 and 202.214.0.0/16 ranges, which are registered under e-society’s ASN (AS131800). The crawler respects HTTP/1.1 Keep-Alive connections and sends a User-Agent string exactly as specified in its official announcement. It performs both GET and HEAD requests, and follows 301/302 redirects up to a depth of 5 hops. The bot does not execute JavaScript or parse dynamic content, focusing exclusively on static HTML and robots.txt-accessible resources.

📋 robots.txt Compliance

Based on the official documentation and network monitoring reports from John Graham-Cumming’s blog (2024), e-societyrobot fully honors Disallow directives in robots.txt and also respects Crawl-Delay directives when present. The company explicitly states that they check robots.txt before each crawl session and will not access paths excluded byDisallow rules. However, the bot does not support the Allow directive for overriding disallowances, adhering strictly to the original robots.txt specification (RFC 9309).

🔍 Detection Indicators

The definitive User-Agent string is e-societyrobot/1.0, with a variant e-societyrobot/2.0 beginning in October 2024. The bot also sends a custom HTTP header X-Robot-ID: e-societyrobot to help webmasters identify it. Additional behavioral fingerprints include the consistent use of Accept: text/html,application/xhtml+xml and Accept-Language: en,ja;q=0.9. Its request pattern shows a fixed 1-second interval between batches to reduce server load.

📊 Data Usage

Collected data is processed and tokenized to train e-society’s foundational models, including the eGPT-4 series. According to e-society’s privacy policy at https://www.e-society.co.jp/privacy/, the company retains crawled content for a maximum of 90 days after training, then strips all personally identifiable information before storing in anonymized feature stores. The data is not sold to third parties but is used exclusively for internal AI research and product development.

⚙️ Rate Limiting Policy

e-societyrobot is rate-limited because its 10–20 requests per second can overwhelm smaller servers if unrestricted. The policy of threshold-based blocking—allowing up to 500 requests per minute per IP before a temporary ban—ensures the bot remains courteous while still collecting sufficient data for model improvement without causing denial-of-service conditions.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.