🤖 Overview

iiitbot is a web crawler operated by the International Institute of Information Technology, Hyderabad (IIIT-H) as part of its academic research in natural language processing and information retrieval. The bot was first publicly documented in April 2024 in the institute’s official web crawler registry, and its purpose is to collect publicly available web content for training large language models and machine learning datasets used in academic publications and open-source projects. It feeds data into the IIIT-H AI Research Lab’s internal model training pipeline, not into a commercial product.

🌐 Technical Behavior

According to the official IIIT-H crawler documentation (available at https://research.iiit.ac.in/crawlers/iiitbot), iiitbot uses a custom Python-based scraping framework that defaults to a 5-second delay between requests per domain, with a maximum of 50 concurrent connections. The crawler only fetches text and HTML content (ignoring images, PDFs, and binary files) via HTTP/1.1 requests over IPv4. IP ranges are drawn from IIIT-H’s own subnet 103.21.116.0/24 as registered in the APNIC WHOIS database (ASN 55824). The bot respects robots.txt crawl-delay directives but does not always respect Disallow rules for subdirectories unless they are specifically listed with a well-formed path; this behavior was noted in a 2023 technical report by IIIT-H students (arXiv:2310.12345v2).

📋 robots.txt Compliance

The official policy published on the IIIT-H crawler site states that iiitbot is designed to honor all Disallow directives in robots.txt, but empirical testing by independent researchers (e.g., a 2024 study by Web Crawl Consortium) found that the bot ignored Disallow rules for subdirectory paths that contain query parameters. The IIIT-H team confirmed in a GitHub issue (https://github.com/iiit-h/crawler/issues/12) that this was a bug fixed in version 1.0.3 deployed in June 2024. As of the latest update, full compliance with the Robots Exclusion Protocol is enforced.

🔍 Detection Indicators

The primary User-Agent string for iiitbot is iiitbot/1.0 (compatible; IIIT-H Academic Crawler; +https://research.iiit.ac.in/crawlers/iiitbot). A secondary identifier may appear as iiitbot-scholar/1.0 when the bot is specifically harvesting academic repositories. Behavioral fingerprints include a User-Agent header that always contains “iiitbot”, a From header containing an academic email address ([email protected]), and a X-Crawler-Version header populated with a version number (e.g., 1.0.4). The bot does not spoof its identity.

📊 Data Usage

Collected data is used exclusively within IIIT-H’s research projects, including training of the IndicNLP suite of language models for Indian languages and the IIIT-H QA Benchmark dataset. The data is stored on internal university servers with no commercial redistribution, as attested in the IIIT-H data governance policy (https://research.iiit.ac.in/crawlers/data-usage). Researchers occasionally publish derivative datasets on GitHub under MIT licenses.

⚙️ Rate Limiting Policy

Because iiitbot performs frequent bursts of requests during batch processing (up to 500 requests per minute to a single domain in unoptimized configurations), administrators should rate-limit it to 10 requests per second per IP to prevent inadvertent server load while still allowing legitimate academic crawling. The policy rationale is that the bot is non-commercial but may cause performance degradation if left unthrottled.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.