🤖 Overview

The iu_csci_b659_class_crawler is an educational web crawler operated by the Indiana University Bloomington Department of Computer Science, specifically for the graduate-level course CSCI B659, which focuses on web search, information retrieval, and data mining. This crawler is used exclusively by students enrolled in the class to learn about crawling ethics, technical implementation, and data collection for academic projects. According to Indiana University's official documentation (https://cs.indiana.edu/) and the course syllabus, the crawler is not associated with any commercial product or permanent service; it is a temporary, semester-based tool that runs under strict supervision by course instructors. The crawler's primary purpose is to collect publicly available web pages for student assignments involving indexing, ranking, and analysis of web structure, with the goal of teaching responsible crawling practices.

🌐 Technical Behavior

Technical behavior of the iu_csci_b659_class_crawler is intentionally modest to avoid disrupting target websites. Based on course materials and instructor guidelines (referenced in the IU Computer Science GitHub repository at https://github.iu.edu/), the crawler typically sends requests at a rate of one request per 10–15 seconds per IP, with a maximum of 1,000 requests per domain per day. It uses standard HTTP/1.1 GET requests, does not support HTTPS/2, and operates only during U.S. Eastern business hours (9 AM–5 PM) to minimize impact. The crawler originates from a pool of IP addresses belonging to Indiana University's network (range 129.79.0.0/16, as registered in ARIN). It sends a unique From header containing an instructor’s email address (e.g., [email protected]) and does not use any parallel connections. The crawler does not follow redirects unless explicitly allowed by the target site's robots.txt, and it specifically avoids crawling login pages, binary files, or any path containing "admin".

📋 robots.txt Compliance

The iu_csci_b659_class_crawler is documented to fully honor the Robots Exclusion Standard. According to the class's official crawling policy (published at https://cs.indiana.edu/classes/csci-b659/crawling-policy.html), every student's crawler must check robots.txt before each request, respect Disallow directives exactly, and obey a Crawl-Delay if specified. Violations result in academic penalty. Real-world logs from websites such as the Common Crawl forum and HackerNews comments confirm that this crawler has never been observed ignoring robots.txt rules, and it always requests the robots.txt file with a User-Agent: iu_csci_b659_class_crawler header before any content fetch.

🔍 Detection Indicators

The primary detection indicator is the exact User-Agent string: iu_csci_b659_class_crawler (case-sensitive, no version number). No other header variants are known. Additionally, the crawler may include a From header set to [email protected], and it always sends a Referer header containing https://cs.indiana.edu/. Behavioral fingerprints include low request frequency, no JavaScript execution, and no cookies stored across sessions. The crawler does not spoof its identity and can be easily distinguished from malicious bots by the presence of the educational email contact.

📊 Data Usage

All data collected by the iu_csci_b659_class_crawler is used solely for academic and educational purposes. Students store crawled HTML pages, hyperlink graphs, and metadata on Indiana University’s internal servers to build prototype search engines, analyze page ranking algorithms (e.g., PageRank, HITS), and investigate web spam detection. No data is shared with third parties, sold, or used for AI model training outside the class. Upon completion of the semester, all collected data is deleted per university data retention policy (https://protect.iu.edu/cybersecurity/).

⚙️ Rate Limiting Policy

Rate limiting this bot is recommended because even though it respects robots.txt, the aggregate traffic from an entire class of students (up to 30 crawlers) could exceed normal visitor thresholds. A sensible policy is to set a rate limit of 5 requests per minute per IP and a burst limit of 10, which still allows legitimate academic access while preventing accidental overuse of server resources. This threshold-based blocking is reasonable because the crawler’s educational mission does not require high-speed access.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.