bottybot
Bottybot is a web crawler operated by Botty AI Inc., a company specializing in large-scale data collection for artificial intelligence model training. First publicly documented in early 2024, bottybot systematically gathers publicly available web pages to feed into Botty's proprietary language models and retrieval-augmented generation (RAG) systems. Its primary purpose is to expand the training corpus for next-generation AI assistants and search engines. According to Botty AI’s official documentation (botty.ai/crawler), the bot operates transparently and is designed to comply with webmaster preferences.
Bottybot employs a headless Chromium-based engine to render JavaScript-heavy pages before extraction, mimicking real user browsing sessions. It crawls with an average request rate of 5–10 requests per second per originating IP, but can burst up to 30 requests per second during initial site discovery. IP ranges are primarily from Amazon Web Services (AWS) (e.g., 52.48.0.0/16, 54.77.0.0/16) and occasionally from Google Cloud Platform. The crawler uses HTTPS/2 by default and advertises a Accept-Language: en-US,en;q=0.5 header. Bottybot supports conditional GET requests via ETags and Last-Modified headers to avoid re-fetching unchanged content. A GitHub repository (github.com/botty-ai/bottybot-tools) references its crawl pattern: it prioritizes pages with high link equity and sitemap URLs, then follows internal links up to a depth of 4.
Bottybot fully honors robots.txt directives, including Disallow and Crawl-delay instructions. The official Botty AI FAQ states that “bottybot will never ignore a webmaster’s robots.txt rule.” Evidence from real server logs (e.g., Cloudflare AI Bot Management reports) confirms that bottybot ceases crawling any path prefixed with /wp-admin, /private, or other blocked directories within seconds of encountering a Disallow line. The crawler also respects Allow directives when specified.
The primary User-Agent string is Mozilla/5.0 (compatible; bottybot/1.0; +https://botty.ai/bot). A secondary string bottybot/2.0 (compatible; +https://botty.ai/bot) is used for newer deployments. Behavioral fingerprints include a persistent X-Botty-Crawl-ID header containing a UUID, and a Referer: https://botty.ai/ header on most requests. The bot never sends a Cookie header and ignores Set-Cookie responses. Cloudflare’s bot management API classifies bottybot as a “verified bot” with a confidence score >99%.
Collected content is stored in Botty AI’s BIndex – a vector database used to train large language models (LLMs) and improve the accuracy of answer generation in the Botty search product. According to Botty AI’s privacy policy (botty.ai/privacy), data is anonymized before being ingested into training sets, and no personal identifiable information is retained for model fine-tuning. The data also feeds into a public API for academic research under the “Botty Open Corpus” initiative announced in August 2024.
Bottybot is rate-limited because its default crawl rate can overwhelm smaller servers and shared hosting environments. Standard practice is to impose a threshold of 100 requests per minute per IP using tools like fail2ban or web application firewall rules. This policy ensures fair resource allocation while still allowing Botty AI to index essential public content. Botty AI recommends webmasters set a Crawl-delay: 10 in robots.txt to reduce load if needed.
🛡️
Boteraser automatically detects and blocks unwanted bots — protecting your site from scrapers, DDoS bursts, and credential stuffing attacks without slowing down real visitors.
✅ Start Free ProtectionSetup takes under a minute · Free trial available
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
Stay up to date with the latest from Boteraser.
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
These cookies are needed for adding comments on this website.
These cookies are used for managing login functionality on this website.
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
You can find more information in our Cookie Policy and Privacy Policy.