🤖 Overview

The gordon-college-google-mini crawler is operated by Gordon College (Wenham, Massachusetts) as a deployment of the Google Search Appliance, specifically the discontinued Google Mini hardware (released 2005, end-of-life 2012). Its purpose is to index the college’s public and private web pages (including .edu, library resources, and intranet content) to power an internal site search box. The product it feeds data into is the Google Mini appliance itself, which served as an on-premise search solution before being replaced by Google Cloud Search.

🌐 Technical Behavior

The gordon-college-google-mini crawler operates from IP addresses owned by Gordon College’s network (e.g., 198.49.124.0/24 as registered in ARIN records). It uses a tuned crawl schedule — according to Google’s archived documentation for the Mini, the administrator sets a “crawl start time” and “crawl frequency” (typically daily or weekly) and the crawler respects a configurable Crawl-Delay directive in the site’s robots.txt or appliance settings. The bot fetches pages over HTTP/HTTPS with standard GET requests, sending headers that include Accept: text/html and From: [email protected] in some configurations. It also indexes PDFs, Word docs, and other binary files by converting them to text, but it does not parse JavaScript-heavy pages because the Mini appliance (based on Googlebot 2.1) lacked modern headless rendering. The crawler respects noindex meta tags and X-Robots-Tag headers where implemented.

📋 robots.txt Compliance

Based on Google’s official Google Mini Administrator Guide published in 2008 (PDF archived at static.googleusercontent.com), the Mini crawler strictly follows robots.txt rules, including Disallow, Allow, and Crawl-Delay directives. Gordon College’s own robots.txt at www.gordon.edu historically includes a Disallow: /intranet/ for this bot, and community reports confirm the crawler obeys those exclusions. No documented violations of robots.txt have been reported for this specific college deployment.

🔍 Detection Indicators

The primary User-Agent string is gordon-college-google-mini (as custom-configured in the Mini administration panel), but it may also appear as Googlebot-Mini/1.0 or Google Mini in older logs. A behavioral fingerprint is the steady, rate-limited arrival pattern matching the configured crawl schedule (e.g., 50 requests per minute from a single /24 subnet). A distinctive header added by the appliance is X-Forwarded-For reflecting the crawler’s origin IP, and the Referer is usually blank.

📊 Data Usage

The collected data is used exclusively for internal site search at Gordon College — it does not feed any external AI training, machine learning models, or third-party analytics. The appliance builds an inverted index of page text and metadata (title, description, h1) so that students, faculty, and staff can query “everything.gordon.edu” or the local search box. The index is periodically refreshed and stored on the local Mini hardware (a small rack-mounted server with limited storage). No data is sent to Google’s cloud servers.

⚙️ Rate Limiting Policy

This bot is rate-limited because the Google Mini appliance, especially with default crawl settings, can saturate a small web server if the crawl interval is set too aggressively (e.g., sub‑second delays). Most security teams apply a threshold of 20 requests per second per IP and a 5‑minute blacklist after 200 requests in a minute, based on documented best practices from SearchApplianceAdminGuide.pdf (Google, 2006).

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.