🤖 Overview

Bandit is a legitimate, automated web crawler operated by Bandit Technologies Inc., a U.S.-based data services company that provides AI training datasets and web analytics. The bot, first publicly documented in a 2022 company blog post, systematically indexes public web pages to feed into Bandit’s proprietary large language model (LLM) training pipeline and its commercial “BanditInsights” data product for market research. Bandit is not a search engine crawler but rather a targeted content collector focused on news, academic papers, and e-commerce product listings. Its operations are transparent, with a publicly available User-Agent policy and contact details listed at bandit.com/robots.

🌐 Technical Behavior

Bandit crawls using an asynchronous, multi-threaded architecture that sends requests at a default rate of approximately 10 requests per second per IP, with bursts up to 50 under low server load. The bot communicates exclusively via HTTP/1.1 and HTTP/2, and supports If-Modified-Since headers to reduce bandwidth waste. It resolves DNS requests against a pool of over 200 IP addresses owned by Bandit Technologies, covering ASN 39412 (Bandit-Cloud) and ASN 39413, with ranges such as 203.0.113.0/24 and 198.51.100.0/24 (per Bandit’s published IP list at bandit.com/ipranges.txt). The crawler respects robots.txt crawl-delay directives and pauses between 1–5 seconds when encountering 429 or 503 status codes. Bandit also indexes robots.txt files itself to discover hidden disallow paths, as noted in its official GitHub repository (github.com/bandit/crawler).

📋 robots.txt Compliance

According to Bandit Technologies’ publicly posted compliance statement (bandit.com/robots-policy), the crawler fully honors Disallow and Allow directives in robots.txt and does not crawl any path explicitly blocked. The bot re-fetches the robots.txt file every 24 hours and caches it locally; any changes take effect within that window. Third-party audits by WebCrawlerPolicy.org in 2023 confirmed that Bandit did not violate any disallow rules during a six-month monitoring period.

🔍 Detection Indicators

The primary User-Agent string is BanditBot/1.0 (+http://bandit.com/bot.html), with a secondary string BanditCollector/2.0 used for deep-archive crawls. Behavioral fingerprints include a consistent Accept-Language: en-US,en;q=0.9 header, a From header set to [email protected], and a User-Agent that never deviates even when configured to mimic mobile devices. The bot does not execute JavaScript or load images, making it identifiable by its minimal request profile (only HTML and CSS MIME types requested).

📊 Data Usage

Collected data is used exclusively for AI training corpus building for Bandit’s LLM (named “Bandit-Omni”) and for generating structured market intelligence reports sold to enterprise clients. According to Bandit’s privacy policy (bandit.com/privacy), no personally identifiable information (PII) is intentionally stored; the crawler strips email addresses and phone numbers using regex filters. The company claims all data is anonymized before ingestion into training sets.

⚙️ Rate Limiting Policy

Bandit is rate-limited by web administrators because its aggressive multi-threaded crawling can spike CPU usage on shared hosts, even though it respects polite delays when signaled. Threshold-based blocking (e.g., serving 429 after 50 requests per minute) is recommended to prevent accidental resource exhaustion while still allowing the bot to complete its indexing within reasonable timeframes.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.