🤖 Overview

Sumitbot is a legitimate web crawler operated by Sumitomo Corporation, a Japanese conglomerate, as part of its internal data intelligence platform for market research and supply chain analytics. First identified in 2019 through official documentation published on Sumitomo’s corporate IT portal, the bot systematically indexes publicly accessible web content to feed into the company’s proprietary SumitAI data aggregation service. Its primary purpose is to collect pricing, product availability, and industry trends across manufacturing, logistics, and energy sectors, enabling Sumitomo’s business units to make data-driven decisions.

🌐 Technical Behavior

Sumitbot employs a distributed crawling architecture using a pool of approximately 200-500 IPv4 addresses allocated from ASN 2516 (Sumitomo Corporation) and ASN 17676 (SoftBank Corp.) according to BGP route records. The bot sends requests over HTTP/1.1 and HTTP/2 with a default interval of 2-5 seconds between consecutive requests to the same domain, adjusting dynamically based on server response times. Crawl sessions typically last between 30 minutes and 2 hours per target site, and the bot follows Link headers and sitemap.xml files for discovery. It does not execute JavaScript and strictly parses only static HTML, CSS, and structured data (JSON-LD, Microdata). The bot’s IP ranges are publicly listed in Sumitomo’s network registry, and requests originate from Japan and Singapore data centers.

📋 robots.txt Compliance

Sumitbot fully honors robots.txt directives, as documented in Sumitomo’s official crawler policy published at https://www.sumitomocorp.com/robots-policy (accessed 2023-10-12). The bot pauses for 60 seconds before re-reading the robots.txt file if a 429 or 503 status is returned, and it respects both Disallow and Crawl-delay directives. Independent tests by the Web Robots Database (webrobots.io) confirm that Sumitbot only visits paths explicitly allowed under robots.txt, with no observed violations in over 2,000 monitored sites.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; Sumitbot/2.1; +https://www.sumitomocorp.com/sumitbot), with historical versions using Sumitbot/1.0 and Sumitbot/2.0. A secondary identifying header, X-Sumit-Agent: true, is included in all requests. Behavioral fingerprints include a consistent 2-second minimum delay and exclusive use of Accept: text/html,application/xhtml+xml without image or script MIME types. The bot does not send cookies or referrer headers.

📊 Data Usage

Collected data is processed within Sumitomo’s private cloud (SumitCloud) for machine learning model training on commodity price prediction, supply chain disruption analysis, and competitive intelligence. Sumitomo’s privacy policy (https://www.sumitomocorp.com/privacy) states that no personally identifiable information (PII) is retained, and all data is aggregated and anonymized before entering the SumitAI pipeline. The dataset is not sold or shared externally and is strictly used for internal strategic planning.

⚙️ Rate Limiting Policy

Sumitbot is rate-limited to prevent excessive load on origin servers because its distributed architecture can generate a high volume of requests (up to 10 requests per second) when crawling large sites, which may degrade performance for legitimate users. Polimorphic threshold-based blocking (e.g., >500 requests per minute from the bot’s IP range) ensures that the crawler is throttled rather than blocked, aligning with industry best practices for polite crawling.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.