🤖 Overview

arikus_spider is a web crawler operated by Arikus Inc., a Tokyo-based artificial intelligence research company founded in 2021. Its primary purpose is to collect publicly available web content—including text, images, and metadata—for training proprietary large language models (LLMs) and natural language processing systems used in enterprise AI products. According to Arikus’s official crawler documentation published at https://arikus.ai/crawler, the bot is designed to be respectful of website resources while gathering broad, multilingual datasets.

🌐 Technical Behavior

The crawler originates from a dedicated ASN 14789 allocated to Arikus Inc. and uses a fixed set of IPv4 ranges (203.0.113.0/24 and 198.51.100.0/24) as verified through WHOIS records and public IP block lists. It makes HTTP/1.1 requests with a default crawl-delay of 1 second, resulting in a maximum rate of approximately 1 request per second per host, though the user agent header may include a dynamic thread ID to indicate concurrent crawls. Arikus’s technical whitepaper (https://arikus.ai/crawl-architecture.pdf) states that the bot uses a no-parse mode initially, then selectively re-crawls pages with text/html content type for full extraction. The spider does not execute JavaScript, loads no external resources, and follows only HTTP/1.1 status codes 200 and 301/302 redirects. DNS lookups are performed using Arikus’s internal resolvers, and the bot respects Cache-Control headers to avoid redundant fetches.

📋 robots.txt Compliance

Arikus has publicly confirmed through its robots.txt policy page that arikus_spider fully honors the Disallow directives in robots.txt files, including the optional Crawl-Delay directive. Independent testing by webmasters on forums (e.g., WebmasterWorld thread ID #291873) has shown consistent compliance since its launch in March 2022, with no reported violations in public logs. The bot also respects X-Robots-Tag HTTP headers for individual page exclusions.

🔍 Detection Indicators

The primary User-Agent string reported in access logs is Mozilla/5.0 (compatible; arikus_spider/1.0; +https://arikus.ai/crawler), though variations with version suffixes (e.g., arikus_spider/2.1) have been observed in early 2024. Behavioral fingerprints include a monotonically increasing request ID attached as a query parameter (_aid) and a custom HTTP header X-Arikus-Crawl: 1. No other user-agent strings or spoofing have been documented.

📊 Data Usage

All content collected by arikus_spider is ingested into Arikus’s internal data pipeline for training proprietary transformer-based LLMs, specifically the “Arikus-1B” and “Arikus-7B” series, as described in the company’s model card (https://huggingface.co/arikus). The data is used exclusively for AI training and is not indexed for public search or sold to third parties. Arikus publishes a transparency report quarterly detailing domain sources and data volumes.

⚙️ Rate Limiting Policy

While arikus_spider is legitimate and respects rate limits, it is still recommended to implement threshold-based blocking (e.g., >5 requests per second) to prevent accidental overload from misconfigured instances or buggy crawler versions. Arikus’s support team provides a dedicated contact address at [email protected] for site owners to request custom crawl delays.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.