Skip to main content

Boteraser | Website and Server Security Solutions

bottybot

Bot User-Agent: bottybot

🤖 Overview

Bottybot is a web crawler operated by Botty AI Inc., a company specializing in large-scale data collection for artificial intelligence model training. First publicly documented in early 2024, bottybot systematically gathers publicly available web pages to feed into Botty's proprietary language models and retrieval-augmented generation (RAG) systems. Its primary purpose is to expand the training corpus for next-generation AI assistants and search engines. According to Botty AI’s official documentation (botty.ai/crawler), the bot operates transparently and is designed to comply with webmaster preferences.

🌐 Technical Behavior

Bottybot employs a headless Chromium-based engine to render JavaScript-heavy pages before extraction, mimicking real user browsing sessions. It crawls with an average request rate of 5–10 requests per second per originating IP, but can burst up to 30 requests per second during initial site discovery. IP ranges are primarily from Amazon Web Services (AWS) (e.g., 52.48.0.0/16, 54.77.0.0/16) and occasionally from Google Cloud Platform. The crawler uses HTTPS/2 by default and advertises a Accept-Language: en-US,en;q=0.5 header. Bottybot supports conditional GET requests via ETags and Last-Modified headers to avoid re-fetching unchanged content. A GitHub repository (github.com/botty-ai/bottybot-tools) references its crawl pattern: it prioritizes pages with high link equity and sitemap URLs, then follows internal links up to a depth of 4.

📋 robots.txt Compliance

Bottybot fully honors robots.txt directives, including Disallow and Crawl-delay instructions. The official Botty AI FAQ states that “bottybot will never ignore a webmaster’s robots.txt rule.” Evidence from real server logs (e.g., Cloudflare AI Bot Management reports) confirms that bottybot ceases crawling any path prefixed with /wp-admin, /private, or other blocked directories within seconds of encountering a Disallow line. The crawler also respects Allow directives when specified.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; bottybot/1.0; +https://botty.ai/bot). A secondary string bottybot/2.0 (compatible; +https://botty.ai/bot) is used for newer deployments. Behavioral fingerprints include a persistent X-Botty-Crawl-ID header containing a UUID, and a Referer: https://botty.ai/ header on most requests. The bot never sends a Cookie header and ignores Set-Cookie responses. Cloudflare’s bot management API classifies bottybot as a “verified bot” with a confidence score >99%.

📊 Data Usage

Collected content is stored in Botty AI’s BIndex – a vector database used to train large language models (LLMs) and improve the accuracy of answer generation in the Botty search product. According to Botty AI’s privacy policy (botty.ai/privacy), data is anonymized before being ingested into training sets, and no personal identifiable information is retained for model fine-tuning. The data also feeds into a public API for academic research under the “Botty Open Corpus” initiative announced in August 2024.

⚙️ Rate Limiting Policy

Bottybot is rate-limited because its default crawl rate can overwhelm smaller servers and shared hosting environments. Standard practice is to impose a threshold of 100 requests per minute per IP using tools like fail2ban or web application firewall rules. This policy ensures fair resource allocation while still allowing Botty AI to index essential public content. Botty AI recommends webmasters set a Crawl-delay: 10 in robots.txt to reduce load if needed.

🛡️

Stop Bots. Save Bandwidth. Protect Revenue.

Boteraser automatically detects and blocks unwanted bots — protecting your site from scrapers, DDoS bursts, and credential stuffing attacks without slowing down real visitors.

✅ Start Free Protection

Setup takes under a minute  ·  Free trial available

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.

✓