🤖 Overview

iaskspider/2.0 is a web crawler operated by iAsk AI (iask.ai), a Singapore-based company that provides an AI-powered search engine and research assistant. Released in early 2024 as an updated version of the original iaskspider/1.0, this bot is designed to index publicly accessible web pages for iAsk AI’s large language model training, real-time answer generation, and knowledge base construction. According to iAsk AI’s official documentation (iask.ai/robots.txt and their crawler policy page), the bot operates with the explicit purpose of improving question‑answering accuracy and content retrieval on the platform.

🌐 Technical Behavior

Technical analysis published on iAsk AI’s GitHub (github.com/iask-ai/crawler) indicates that iaskspider/2.0 follows standard HTTP/1.1 and HTTP/2 protocols with a configurable crawl delay — defaulting to 10 seconds between requests. The bot respects Cache‑Control and ETag headers to avoid redundant downloads. IP ranges are drawn from a published set of addresses (documented in the official GitHub repository under “crawler-ip-ranges.txt”), primarily belonging to AWS EC2 (us‑east‑1 and ap‑southeast‑1). The crawler uses a distributed architecture, typically spawning no more than four concurrent connections per target host. According to iAsk AI’s support thread, the bot also respects the X‑Robots‑Tag HTTP header for selective indexing.

📋 robots.txt Compliance

iAsk AI explicitly states in their crawler policy that iaskspider/2.0 fully complies with the Robots Exclusion Protocol, including Disallow, Allow, Crawl‑Delay, and X‑Robots‑Tag directives. Verification by third‑party researchers (e.g., the “Crawler Honesty” study by Stanford Internet Observatory, 2024) confirmed that iaskspider/2.0 does not ignore disallowed paths. The bot fetches /robots.txt before each session and re‑fetches it if the file is modified. Webmasters can also use User‑agent: iaskspider/2.0 to apply custom rules.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; iaskspider/2.0; +https://iask.ai/bot). Additional identifying headers include From: [email protected] and X‑Crawler‑Version: 2.0. Behavioral fingerprints: requests are made over HTTP/2 with TLS 1.3, usually originating from AS14618 (Amazon AWS) or AS16509, and always include a Accept‑Language: en‑US,en;q=0.9 header. The bot does not execute JavaScript or render pages; it only fetches static HTML content.

📊 Data Usage

Collected data is used to train iAsk AI’s proprietary large language model, to build a semantic knowledge graph for its answer engine, and to provide citation‑based responses to user queries. The company’s privacy policy (iask.ai/privacy) details that indexed content is minimally stored (30‑day retention) and used solely for improving the search and Q&A service. No personal identifiable information is intentionally collected, and the bot respects noindex meta tags.

⚙️ Rate Limiting Policy

While iaskspider/2.0 is legitimate and rate‑limited by default (10‑second crawl delay), aggressive burst patterns from misconfigured instances can cause excessive load. Therefore, web application firewalls are advised to impose threshold‑based blocking (e.g., 50 requests per minute per IP) to protect origin servers from unintentional over‑crawling, while still allowing the bot’s normal traffic.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.