Skip to main content

Boteraser | Website and Server Security Solutions

sokitomi crawl

Crawler User-Agent: sokitomi-crawl

🤖 Overview

sokitomi crawl is a web crawler operated by SOKitomi Inc., a Japanese company, designed to collect publicly accessible web data for the purpose of training and improving large-language models (LLMs) and AI systems developed by the company. Its stated goal is to enable high-quality AI training while respecting website policies.

🌐 Technical Behavior

The crawler identifies itself via the User-Agent string sokitomi-crawl/1.0 and typically requests pages using HTTP/1.1 with standard GET requests. It can operate both in single-page and recursive crawl modes, respecting standard HTTP Crawl-Delay directives when present. IP ranges are dynamically assigned and often originate from Japanese cloud providers, though the exact ranges are not publicly documented. The crawler may appear with varying frequency, typically a few requests per second per target domain, to avoid overwhelming servers. It does not support JavaScript rendering or cookies, focusing purely on static HTML content extraction.

📋 robots.txt Compliance

Documentation from SOKitomi explicitly states that sokitomi-crawl honors Disallow directives in robots.txt files. The crawler parses the file at the start of each crawl session and will not access any paths listed under Disallow for its User-Agent. It also respects Crawl-Delay directives if set. This compliance is verified in the official guidance published on the company’s website.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string sokitomi-crawl/1.0, which is case-sensitive. Additionally, the crawler sends a From header containing a contact email (e.g., [email protected]). Requests originate from IP addresses that reverse-resolve to hostnames under *.sokitomi.com. No unique request header beyond the standard HTTP fields is used, but the behavior of ignoring JavaScript makes it distinguishable from browser traffic.

📊 Data Usage

Data collected by sokitomi-crawl is used exclusively to train and improve internal AI models, including natural language understanding and generation systems. The company states that no personal data is intentionally collected, and they deduplicate content to avoid over-representation. Crawled data is not publicly shared or sold.

⚙️ Rate Limiting Policy

Rate limiting for sokitomi-crawl is recommended to prevent excessive resource consumption on shared hosting environments. Its default crawl rate, while moderate, can accumulate load over time, and threshold-based blocking is justified to protect site stability without denying legitimate AI research access.

🛡️

Stop Bots. Save Bandwidth. Protect Revenue.

Boteraser automatically detects and blocks unwanted bots — protecting your site from scrapers, DDoS bursts, and credential stuffing attacks without slowing down real visitors.

✅ Start Free Protection

Setup takes under a minute  ·  Free trial available

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.

✓