🤖 Overview

Mammoth is a web crawler operated by Mammoth AI Inc., a startup that develops large language models for commercial AI products. Its purpose is to collect publicly accessible web content to train and improve the Mammoth series of generative models, first announced in a 2023 blog post on mammoth.ai.

🌐 Technical Behavior

The Mammoth crawler uses a custom asynchronous HTTP client written in Python and deployed across Amazon Web Services EC2 instances in the us‑east‑1 and eu‑west‑1 regions. It performs requests at an average of 15 requests per second per IP, with bursts up to 40 rps during initial discovery. The crawler follows links recursively up to a depth of 10, respecting a mandatory crawl delay of 5 seconds between consecutive requests to the same origin. IP ranges come from AWS allocations such as 3.0.0.0/9 and 52.0.0.0/15, and it negotiates HTTP/2 when supported. The default Accept header is ‘text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8’.

📋 robots.txt Compliance

Official documentation at https://mammoth.ai/crawler-policy states that Mammoth fully honors robots.txt Disallow directives and also respects Crawl-Delay if specified. It parses robots.txt before each new crawl session and caches the rules for 24 hours. Independent testing by the Webmaster Community (2024) confirmed compliance with no violations reported.

🔍 Detection Indicators

The primary identifier is the User‑Agent string ‘Mammoth/1.0 (+https://mammoth.ai/bot)’. Behavioral fingerprints include a consistent order of requests: robots.txt first, then a rapid burst of pages followed by a 5‑second pause. Reverse DNS lookups resolve to hostnames like ec2‑*.compute‑1.amazonaws.com. An optional custom header ‘X‑Mammoth‑Bot: true’ may be present.

📊 Data Usage

Collected data is deduplicated, filtered to remove personally identifiable information, and then used to train the Mammoth family of language models, which are deployed as commercial APIs and enterprise products. The company publishes a transparency report annually at mammoth.ai/transparency detailing data sources and processing steps.

⚙️ Rate Limiting Policy

Threshold‑based rate limiting is recommended because Mammoth’s burst traffic can overwhelm smaller servers. A limit of 100 requests per minute per IP protects server resources while allowing the legitimate crawler to complete its work without being blocked entirely.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.