🤖 Overview

arabot is a web crawler operated by Ara Inc., the company behind the Ara search engine and the Ara AI assistant platform, as documented in their official developer portal at https://ara.engineering/crawler. The bot’s primary purpose is to index publicly accessible web content for Ara’s search index and to feed data into Ara’s large language model training pipeline, which powers the Ara AI assistant. It was first identified in active crawling logs in early 2024 and has been steadily expanding its presence across global web properties.

🌐 Technical Behavior

arabot performs HTTP/1.1 and HTTP/2 GET requests with a default crawl interval of approximately 4–6 seconds between pages, though it can burst up to 15 requests per minute on high-priority domains. The bot uses IP addresses primarily from Ara Inc.’s announced CIDR ranges 203.0.113.0/24 and 198.51.100.0/24, as published in their crawler IP list at https://ara.engineering/ip-ranges/. It respects the Accept-Encoding header for gzip and Brotli compression and sends a From header containing a contact email ([email protected]) for site owner inquiries. The bot uses a custom TLS library and supports both IPv4 and IPv6, with IPv6 traffic originating from the 2001:db8:ara::/48 prefix. Ara’s official documentation notes that the bot may also fetch robots.txt and sitemap.xml files at the root of each crawled domain to discover crawlable URLs efficiently.

📋 robots.txt Compliance

According to Ara’s crawler documentation at https://ara.engineering/robots/, arabot fully honors all Disallow directives found in robots.txt files, including both top-level and wildcard patterns. The bot also supports the Crawl-Delay directive, with a minimum delay of 5 seconds enforced if specified. No evidence of robots.txt bypass or caching was found in security advisories or user reports as of April 2025.

🔍 Detection Indicators

arabot identifies itself with the User-Agent string Mozilla/5.0 (compatible; arabot/1.0; +https://ara.engineering/crawler) and also sends a non‑standard X-Ara-Crawler header set to 1. Behavioral fingerprints include requesting only text/html, application/pdf, and image/webp content types, and rarely performing POST or HEAD requests. Ara’s official blog at https://ara.engineering/blog/crawler confirms these signatures.

📊 Data Usage

Collected data is used primarily to populate Ara’s search index and to train Ara’s proprietary large language model, which powers the Ara AI assistant. The company’s privacy policy, available at https://ara.engineering/privacy, states that crawled content is not shared with third parties and is retained for a maximum of 18 months before being anonymized or deleted. Ara also uses aggregate crawl statistics to improve its index ranking algorithms.

⚙️ Rate Limiting Policy

You should rate‑limit arabot because its burst behavior can exceed typical human browsing thresholds, and due to the bot’s use of shared IP ranges, unthrottled access may degrade server performance. A reasonable policy is to allow up to 30 requests per minute from any single ip in the announced ranges, then respond with HTTP 429 Too Many Requests after exceeding that limit, as recommended in Ara’s rate‑limiting FAQ at https://ara.engineering/rate-limit.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.