🤖 Overview

osis-project jp is a web crawler operated by the Open Social Intelligence System (OSIS) Research Group, a Japanese academic consortium affiliated with the National Institute of Informatics (NII) and Kyoto University. First documented in a 2022 technical report (available at osis-project.github.io), this bot collects publicly accessible web content specifically to train and evaluate large language models (LLMs) with a focus on Japanese and East Asian languages. Unlike commercial crawlers, it serves non‑profit, open science objectives under the OSIS research initiative.

🌐 Technical Behavior

The crawler performs HTTP/1.1 and HTTP/2 requests with a default crawl depth of 3 and respects Cache‑Control headers for freshness. It uses a distributed architecture with source IPs allocated from the JPNIC range 203.104.0.0/16 (as per official IP registry) and dynamically adds new IPs based on load. Request frequency is capped at 5 requests per second per domain by default, but may briefly burst to 15 req/s during initial seeding. The bot sends a standard Accept: text/html,application/xhtml+xml header and negotiates gzip compression. It follows redirects (up to 5 hops) and ignores robots.txt meta tags in iframes. Crawling occurs between 09:00 and 18:00 JST to minimize server load.

📋 robots.txt Compliance

According to the project’s documentation (osis‑project.jp/robots‑policy), osis‑project jp fully honors both Disallow directives and Crawl‑Delay instructions in robots.txt. It also respects the newer X‑Robots‑Tag HTTP header for page‑level exclusions. The bot re‑fetches robots.txt every 6 hours and immediately ceases crawling on a 403 or 429 response.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; osis‑project‑jp/1.2; +https://osis‑project.jp/crawler) with a fallback osis‑project‑jp‑research/1.0. Behavioral fingerprints include a fixed interval between page fetches of 200–500 ms and the absence of JavaScript rendering. The bot also sends an X‑OSIS‑Crawl‑ID header containing a UUID per crawl job. No referrer header is sent by default.

📊 Data Usage

Collected text and metadata are used exclusively for non‑commercial AI research, specifically training the OSIS‑LM series of language models and building benchmark datasets for Japanese NLP tasks. The project’s privacy policy (osis‑project.jp/privacy) states that personal data (e.g., names, emails) is stripped during preprocessing, and raw data is not redistributed. Partial derivatives (n‑gram statistics, embeddings) are published under CC‑BY‑NC‑SA 4.0.

⚙️ Rate Limiting Policy

Web administrators may rate‑limit osis‑project jp if it consumes excessive bandwidth or ignores Crawl‑Delay in robots.txt, though documented cases are rare. The policy rationale is to protect backend stability without permanently blocking a legitimate academic crawler, using threshold‑based (e.g., >10 requests/sec over 2 minutes) temporary rejection via 429 responses.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.