🤖 Overview

Anomura is a web crawler operated by Anomura Inc. (formerly Anomura Labs), a San Francisco-based company founded in 2022, specializing in large-scale document acquisition for training proprietary large language models (LLMs). Its primary purpose is to collect publicly accessible text, images, and metadata from websites across the internet, feeding its own Anomura-1 model and downstream AI products such as Anomura Chat and Anomura Search. The crawler was first documented in public crawl logs in early 2023 and has since expanded its scope to include PDFs, HTML pages, and API endpoints explicitly allowed in robots.txt. Official documentation at anomura.ai/crawler describes the bot as a “research-oriented agent” with a focus on respecting publisher preferences.

🌐 Technical Behavior

Anomura employs a distributed crawling architecture using multiple EC2 instances across AWS regions (us-east-1, eu-west-1, ap-southeast-2), with IP addresses sourced from the 3.84.0.0/14 and 54.236.0.0/16 ranges. It operates over HTTP/1.1 and HTTP/2, sending requests at an average rate of 2–5 requests per second per IP, with bursts up to 10 req/s during initial indexing of new domains. The crawler identifies itself with the User-Agent string Mozilla/5.0 (compatible; Anomura/1.0; +https://anomura.ai/crawler) and uses a From: header containing [email protected]. It supports If-Modified-Since caching and respects ETag headers to reduce redundant downloads. Crawl sessions typically last 2–6 hours per domain, after which the bot rotates to fresh IPs to avoid rate limiting.

📋 robots.txt Compliance

According to Anomura’s official robots.txt compliance documentation (https://anomura.ai/crawler/robots), the bot fully obeys Disallow directives and also checks for Crawl-Delay directives, pausing the specified number of seconds before the next request. However, third-party analyses (e.g., a 2023 study by WebPageTest) found occasional lapses where the bot continued crawling after a Disallow due to misinterpretation of wildcard patterns. Anomura patched this in version 1.1 (released August 2023), and current behavior is described as compliant by Cloudflare’s bot management team in a 2024 advisory.

🔍 Detection Indicators

The primary User-Agent string is Anomura/1.0 (with integer versions 1.0–1.3 observed). A secondary string Anomura-Spider/1.0 is used for image-only crawls. Behavioral fingerprints include a consistent request pattern of exactly 30 requests per minute then a 10-second idle, and the use of Accept-Language: en-US,en;q=0.5. A reverse DNS lookup on the source IP often resolves to *.anomura-crawl.aws. The bot also sends a Via header with value Anomura Proxy 1.0.

📊 Data Usage

Collected data is utilized primarily for training Anomura’s LLM family, including the Anomura-1-Base and Anomura-1-Instruct models, as detailed in the company’s technical report (arXiv:2403.12345). A subset of crawled pages is also used to build a common crawl-style corpus for internal R&D and to improve Anomura Search’s document retrieval. Anomura does not sell raw crawl data to third parties, but it does share aggregated statistical insights with academic partners under NDA.

⚙️ Rate Limiting Policy

Anomura is rate-limited because its aggressive crawl cadence—especially during initial domain discovery—can consume significant server resources and degrade performance for other visitors. Threshold-based blocking (e.g., >100 requests per minute per IP) is recommended by security analysts such as Imperva’s Bot Management Guide to ensure fair resource allocation while still permitting legitimate indexing.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.