sokitomi-crawl
sokitomi crawl is a web crawler operated by SOKitomi Inc., a Japanese company, designed to collect publicly accessible web data for the purpose of training and improving large-language models (LLMs) and AI systems developed by the company. Its stated goal is to enable high-quality AI training while respecting website policies.
The crawler identifies itself via the User-Agent string sokitomi-crawl/1.0 and typically requests pages using HTTP/1.1 with standard GET requests. It can operate both in single-page and recursive crawl modes, respecting standard HTTP Crawl-Delay directives when present. IP ranges are dynamically assigned and often originate from Japanese cloud providers, though the exact ranges are not publicly documented. The crawler may appear with varying frequency, typically a few requests per second per target domain, to avoid overwhelming servers. It does not support JavaScript rendering or cookies, focusing purely on static HTML content extraction.
Documentation from SOKitomi explicitly states that sokitomi-crawl honors Disallow directives in robots.txt files. The crawler parses the file at the start of each crawl session and will not access any paths listed under Disallow for its User-Agent. It also respects Crawl-Delay directives if set. This compliance is verified in the official guidance published on the company’s website.
The primary detection indicator is the User-Agent string sokitomi-crawl/1.0, which is case-sensitive. Additionally, the crawler sends a From header containing a contact email (e.g., [email protected]). Requests originate from IP addresses that reverse-resolve to hostnames under *.sokitomi.com. No unique request header beyond the standard HTTP fields is used, but the behavior of ignoring JavaScript makes it distinguishable from browser traffic.
Data collected by sokitomi-crawl is used exclusively to train and improve internal AI models, including natural language understanding and generation systems. The company states that no personal data is intentionally collected, and they deduplicate content to avoid over-representation. Crawled data is not publicly shared or sold.
Rate limiting for sokitomi-crawl is recommended to prevent excessive resource consumption on shared hosting environments. Its default crawl rate, while moderate, can accumulate load over time, and threshold-based blocking is justified to protect site stability without denying legitimate AI research access.
🛡️
Boteraser automatically detects and blocks unwanted bots — protecting your site from scrapers, DDoS bursts, and credential stuffing attacks without slowing down real visitors.
✅ Start Free ProtectionSetup takes under a minute · Free trial available
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
Stay up to date with the latest from Boteraser.
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
These cookies are needed for adding comments on this website.
These cookies are used for managing login functionality on this website.
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
You can find more information in our Cookie Policy and Privacy Policy.