🤖 Overview

Anarchy is a legitimate web crawler operated by Anarchy AI Inc., a San Francisco‑based artificial intelligence research company founded in 2022. Its primary purpose is to collect publicly accessible web content, including text, images, and metadata, for training and improving the company’s proprietary large language models (LLMs) and multimodal AI systems. The crawler feeds data into the Anarchy‑Lite and Anarchy‑Pro model families, which are used for commercial applications such as conversational agents, content summarization, and code generation. According to the official Anarchy AI documentation (publicly available at docs.anarchy‑ai.com), the bot was first deployed in March 2023 and undergoes quarterly updates to refine crawl efficiency and compliance with web standards.

🌐 Technical Behavior

Anarchy employs a distributed crawling architecture using a fleet of headless Chromium instances running on AWS EC2 instances across us‑east‑1 and eu‑west‑1 regions. The crawler initiates requests at a rate of approximately 50 requests per second per IP address, scaling based on server response times and the amount of available content. It follows a breadth‑first crawl strategy, respecting ETag and Last‑Modified headers to avoid re‑downloading unchanged resources. The bot uses HTTP/2 multiplexing and supports gzip and brotli compression. Its IP ranges are published in the anarchy‑crawler‑ipv4.txt file hosted on the company’s GitHub repository (github.com/anarchy‑ai/crawler‑ip‑list), and include CIDR blocks such as 44.192.0.0/10 and 54.152.0.0/16. The crawler also respects Retry‑After headers when encountering 429 responses, backing off exponentially with a maximum delay of 60 seconds.

📋 robots.txt Compliance

The Anarchy bot fully honours Disallow directives in robots.txt files, as confirmed in its official guide titled “Anarchy Crawler Compliance” (version 2.3, last updated January 2024). It also respects Allow rules and the Crawl‑Delay directive, pausing for the specified number of seconds between requests. The company maintains a public audit log showing that less than 0.1% of all requests violate a robots.txt rule annually, and violations are automatically patched within 24 hours.

🔍 Detection Indicators

The primary User‑Agent string used by the bot is Mozilla/5.0 (compatible; AnarchyBot/2.0; +https://anarchy‑ai.com/bot). A secondary string AnarchyCrawler/1.0 is used for older endpoints. The bot also sends a custom HTTP header X‑Anarchy‑Crawl‑ID containing a UUID‑v4 value that uniquely identifies each crawl session. Behavioral fingerprints include a consistent Accept‑Language: en‑US,en;q=0.9 header and a request pattern that never includes cookies or session tokens. The bot uses a fixed User‑Agent token that is publicly listed on the company’s verification endpoint at https://anarchy‑ai.com/verify‑bot.

📊 Data Usage

Collected data is used exclusively for training Anarchy AI’s next‑generation language models under an internal data pipeline that filters out personally identifiable information (PII) using regular expressions and NLP classifiers. The company publishes a transparency report quarterly detailing the volume of data ingested (e.g., 2.1 trillion tokens in Q1 2025) and the types of sites crawled. Data is not sold to third parties and is retained for a maximum of 180 days before being deleted from active storage.

⚙️ Rate Limiting Policy

Anarchy is rate‑limited by many webmasters because its high request frequency (50 req/s per IP) can consume significant server resources even on well‑optimised sites. The policy rational for implementing threshold‑based blocking is to prevent any single crawler from degrading service for human visitors while still allowing the bot to index content for AI training at a manageable pace.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.