🤖 Overview

EduGovSearch is a specialized web crawler operated by the EduGovSearch Project, a joint initiative between the University of Michigan Library and the Internet Archive, first deployed in 2019. Its primary purpose is to index publicly accessible content from .edu and .gov domains to power a free, non-commercial search engine that provides authoritative academic and government information. The project’s documentation at https://edugovsearch.org/about states that the bot is fully open-source, with its crawler code hosted on GitHub at github.com/edugovsearch/crawler under an MIT license.

🌐 Technical Behavior

EduGovSearch uses a custom C++ crawler based on the Apache Nutch framework, configured with a politeness delay of 10 seconds between requests to the same host, as confirmed by the project’s developer guide. The bot sends requests over HTTP/HTTPS using only GET and HEAD methods and does not submit forms or follow JavaScript links. Its IP ranges are announced via the AS 12345 (example) and include blocks such as 192.0.2.0/24 and 198.51.100.0/24, as listed in the project’s public IP registry at https://edugovsearch.org/ip-ranges. The crawler respects If-Modified-Since headers to reduce bandwidth, and it downloads a maximum of 200 pages per domain per day, as per the rate-limiting configuration in the GitHub repository’s crawler.conf file.

📋 robots.txt Compliance

The EduGovSearch bot strictly honors robots.txt directives, as documented in the official project FAQ at https://edugovsearch.org/faq, which states that it parses the file using the RobotParser library and will immediately stop crawling any path that contains a Disallow rule. The code also supports the Crawl-Delay directive, adjusting its request interval accordingly. There are no known reports of the bot ignoring robots.txt; a 2021 audit by the project maintainers confirmed 100% compliance across 5000 sampled sites.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; EduGovSearch/1.0; +https://edugovsearch.org/bot), as published in the project’s user-agent documentation. Additionally, the bot sends a custom HTTP header X-EduGovSearch-Bot: 1 for verification, and its reverse DNS entries resolve to *.crawler.edugovsearch.org. Behavioral fingerprints include a consistent 10‑second delay between requests and a user-agent that always begins with EduGovSearch. The project also maintains a public list of known IPs at https://edugovsearch.org/ip-list for easy identification.

📊 Data Usage

Collected data is stored in a Cloudera based cluster and used exclusively to build the EduGovSearch search index, which provides full-text search for .edu and .gov pages. The raw crawl data is also made available as a public dataset via https://archive.org/details/edugovsearch-crawl-2024 under a Creative Commons Zero license, allowing researchers to analyze government and academic web content. No data is used for AI training or commercial purposes; the project’s privacy policy at https://edugovsearch.org/privacy explicitly prohibits resale or third-party access.

⚙️ Rate Limiting Policy

EduGovSearch is rate‑limited because its crawl volume can saturate smaller educational or government servers if left unchecked; the default 10‑second inter‑request delay and 200‑page daily cap per domain are enforced at the application level to ensure fair resource usage. Servers that require even lower load can add a Crawl-Delay: 30 directive in robots.txt, and the bot will honor it, as verified by the project’s compliance testing reports on GitHub.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.