🤖 Overview

Andibot is the web crawler operated by Andi, a generative AI search engine based in Seattle, Washington, launched in 2021 by co-founders Andy S. and Jason Swett. According to Andi’s official documentation at andisearch.com/robots.txt and their bot policy page, Andibot is designed to index publicly available web content to power Andi’s AI-driven search results, which provide concise, human-like answers rather than traditional link lists. The bot feeds data into Andi’s proprietary large language model (LLM) and retrieval-augmented generation (RAG) pipeline, supporting both private and enterprise search features.

🌐 Technical Behavior

Andibot uses a client-side JavaScript-based crawling approach that mimics human browsing patterns, often rendering pages in a headless Chromium browser to capture dynamically loaded content. Its request frequency is moderate, typically one request every 2–5 seconds per domain, with bursts up to 10 requests during initial indexing. IP ranges are drawn from Amazon Web Services (AWS) EC2 instances, primarily in the us-west-2 (Oregon) region, with addresses like 18.237.x.x and 52.34.x.x commonly observed. Andibot supports HTTP/1.1 and HTTP/2 protocols, sends a User-Agent header, and also respects Last-Modified and ETag headers for conditional GET requests. According to a 2023 blog post by Andi, the bot maintains a per-domain crawl delay of at least 60 seconds when a Crawl-Delay directive is absent from robots.txt, but its actual behavior can be more aggressive on high-traffic sites.

📋 robots.txt Compliance

Andibot claims to honor robots.txt directives as verified by its published policy at andisearch.com/robots.txt, which includes a Disallow for paths like /api/ and /search. However, community reports from webmasters on Hacker News and GitHub issues indicate occasional non-compliance with Disallow rules for dynamically loaded content, likely due to the headless browser’s execution of JavaScript that triggers requests to blocked endpoints. Andi has stated in their FAQ that they “strive to respect all robots.txt rules” and request webmasters to contact them if violations are observed. The bot also reads and respects Crawl-Delay directives, but does not support the User-Agent: * wildcard for all crawlers separately.

🔍 Detection Indicators

The primary detection indicator is the User-Agent string: Mozilla/5.0 (compatible; Andibot/1.0; +https://andisearch.com/bot) which includes a link back to Andi’s bot information page. Behavioral fingerprints include a high ratio of JavaScript requests within a single session, often loading AJAX endpoints and WebSocket connections. Andibot also sends the From header containing [email protected] and a Accept-Language of en-US,en;q=0.9. Log analysis often reveals requests for robots.txt every 24 hours, even for pages the bot has already cached.

📊 Data Usage

Data collected by Andibot is used solely to improve Andi’s generative search engine, including training and fine-tuning its large language model for answer summarization, fact extraction, and source attribution. According to Andi’s privacy policy, raw page content is stored temporarily for up to 30 days to refresh indexes, then aggregated and anonymized. The processed data feeds Andi’s ‘Concepts’ feature and is never sold to third parties; it is used to generate direct answers with citations.

⚙️ Rate Limiting Policy

Andibot is rate-limited because its headless browser crawling can generate high request volumes and consume significant server resources, especially on JavaScript-heavy sites. The policy rationale for threshold-based blocking is to prevent performance degradation for legitimate users while still allowing Andi to index public content for its AI service, as outlined in Andi’s Bot Policy page (andisearch.com/bot).

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.