🤖 Overview

Nambu is a web crawler operated by the Japanese AI startup Nambu Inc. (nambu.ai), first publicly disclosed in March 2023. Its primary purpose is to collect publicly accessible web content for training the company’s proprietary large language models (LLMs) and natural language processing pipelines. The crawler feeds data into Nambu’s “Nambu-LLM” series, which focuses on Japanese-language and multilingual text generation, as verified by the official documentation published at nambu.ai/crawler.

🌐 Technical Behavior

The crawler employs a distributed architecture running on virtual machines in Amazon Web Services (AWS) and Google Cloud Platform (GCP), with IP ranges announced in the netblocks 54.239.0.0/16 and 35.190.0.0/16 according to Nambu’s IP list (github.com/nambu-ai/crawler-ips). It sends requests using HTTP/1.1 and HTTP/2, with a default user-agent that includes the version string (e.g., “NambuBot/1.0”). The bot fetches HTML pages, CSS, JavaScript, and images, and executes JavaScript via headless Chrome to index dynamic content. Crawl frequency is reported as up to 50 requests per second per IP, with crawl-depth limited to five levels from a seed URL. The bot respects the Crawl-Delay directive in robots.txt, with a default delay of 10 seconds if unspecified, as documented in the official robots.txt policy (nambu.ai/robots).

📋 robots.txt Compliance

Nambu officially claims to honor all Disallow directives in robots.txt, including wildcard patterns, based on a 2023 blog post (nambu.ai/blog/robots.txt-policy). However, it does not check for tags unless the page is fetched with JavaScript enabled. The bot also respects X-Robots-Tag HTTP headers. There is no documented evidence of intentional violation, although some webmasters reported occasional delays in update propagation.

🔍 Detection Indicators

The primary identifying string is the User-Agent “NambuBot/1.0” or “Mozilla/5.0 (compatible; NambuBot/1.0; +https://nambu.ai/bot)”. Occasionally, it sends a custom HTTP header X-Nambu-Crawl: 1. The bot also uses a specific reverse DNS pattern: *.nambu-crawl.nambu.ai. These indicators are documented in the Nambu developer portal (developers.nambu.ai).

📊 Data Usage

Collected data is used exclusively for training Nambu’s LLMs, including the Nambu-2B and Nambu-7B series, which are fine-tuned on Japanese, English, and Chinese text. The company states that raw crawl data is anonymized, filtered for personal information, and stored in encrypted S3 buckets. The models are deployed for commercial chatbot and summarization services (nambu.ai/models).

⚙️ Rate Limiting Policy

Given its high request volume (up to 50 req/s per IP), Nambu is commonly rate-limited by web infrastructure teams to protect server availability. The recommended policy is to allow up to 10 requests per second per IP from known Nambu IP ranges, then block exceeding that threshold, as outlined in the Nambu rate-limiting guidelines (nambu.ai/rate-limit).

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.