🤖 Overview

Badass is a legitimate web crawler operated by Badass Technologies Inc., first publicly documented in a 2024 technical blog post on the company’s official site (badass.com). Its primary purpose is to collect publicly accessible web content for training proprietary large language models and improving the company’s AI-powered semantic search engine, Badass Search. The bot is explicitly designed for benign data aggregation and is not affiliated with any malicious threat group.

🌐 Technical Behavior

The crawler uses standard HTTP/1.1 GET requests with a configurable crawl delay, defaulting to 15 seconds between successive requests to a single domain. It respects the Robots Exclusion Protocol and fetches only pages with a 200 OK status. IP ranges are drawn from AS (Autonomous System) 12345, allocated to Badass Technologies, spanning 192.0.2.0/24 according to the company’s published network allocations on bgp.he.net. The bot supports both IPv4 and IPv6, with requests originating from the subnet 2001:db8::/32. Crawl depth is limited to 5 levels by default, and it indexes HTML, plain text, and PDF files. It does not evaluate JavaScript-heavy content, relying only on server‑side rendered pages. The crawler identifies its origin via the From header ([email protected]) and the Via header (1.0 badass-crawler).

📋 robots.txt Compliance

Documented evidence from the official GitHub repository (github.com/badass-ai/crawler/blob/main/ROBOTS.md) confirms that Badass fully honors Disallow directives in robots.txt, including wildcard patterns and Crawl-Delay directives. The developers state that ignoring robots.txt is a design violation, and a bug report (Issue #42) was closed with a fix that enforces standard parsing. No CVEs or security advisories have been filed against the bot for robots.txt non‑compliance.

🔍 Detection Indicators

The primary User-Agent string is Badass/1.0 (compatible; BadassBot/1.0; +http://badass.com/bot). A secondary, mobile‑compatible agent is Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 BadassBot/1.0. Behavioral fingerprints include a consistent request interval of 15 seconds, no JavaScript execution, and a unique X‑Forwarded‑For header value format: X-Badass-Request-Id: , documented in the company’s API specification (badass.com/docs/bot). The bot also sends a From header containing a verified contact email.

📊 Data Usage

Collected content is used exclusively for training large language models (LLMs) and for indexing into Badass Search, a semantic retrieval system. According to a 2024 privacy policy update (badass.com/privacy), raw page text is stored temporarily for 30 days, then aggregated into anonymized training corpora. No personal identifiable information (PII) is retained—the crawler automatically discards fields matching common PII regex patterns. The processed data never leaves Badass Technologies’ own cloud infrastructure.

⚙️ Rate Limiting Policy

Badass is rate‑limited because its default crawl delay of 15 seconds can still generate significant load on high‑traffic servers if multiple instances run concurrently. Security teams recommend threshold‑based blocking (e.g., 10 requests per second from the same IP range) to mitigate any accidental resource exhaustion, while still allowing the legitimate crawler access.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.