🤖 Overview

Gopher is a historical web crawler operated by the University of Minnesota, originally developed in 1991 to index resources on the Gopher protocol, a precursor to the World Wide Web. It was the first widely-used internet search engine, known as the Gopher Search Engine, and is now maintained as a legacy service by the university’s IT department and volunteers. According to the official Gopher FAQ (gopher://gopher.quux.org:70/0/gopher/faq), the crawler’s purpose is to catalog public Gopher servers (or “gopherspace”) and provide a searchable index via the gopher protocol itself and through web-based gateways.

🌐 Technical Behavior

The Gopher crawler uses a custom TCP-based protocol on port 70, not standard HTTP, to traverse gopherspace. It sends simple text queries (e.g., a bare “ ” followed by a selector string) and parses hierarchical directory listings returned as type 1 (directory) or type 0 (text file) items. Crawling follows a depth-first traversal strategy, with each request manually issued rather than pipelined, resulting in an average request rate of one connection every 3 to 5 seconds. IP address ranges are not publicly documented but are generally within the University of Minnesota’s Class B network (128.101.0.0/16). The crawler does not support HTTP or HTTPS; if a Gopher server is also accessible via HTTP, the crawler will only interact over the native Gopher protocol. When an HTTP server receives a connection on port 70, the crawler may attempt to interpret a standard HTTP response, but it is not designed for that purpose and will likely fail gracefully.

📋 robots.txt Compliance

The Gopher crawler does not implement or obey robots.txt because the Gopher protocol has no equivalent mechanism for resource exclusion. The crawler only follows links within gopherspace and does not parse HTTP robots.txt directives. Some Gopher administrators use a “.cap” file for access control, but this is a server-specific feature, not recognized by the crawler. According to the Gopher Protocol specification (RFC 1436), no standard exclusion mechanism exists, so the crawler expects administrators to disable public access via server configuration if they wish to block indexing.

🔍 Detection Indicators

The Gopher crawler identifies itself with the User-Agent string “Gopher/1.0” or “Gopher/2.0” when connecting to HTTP servers, though such connections are rare. Behavioral fingerprints include connections on TCP port 70 followed by a single line of text (the selector) and a typical lack of HTTP headers. The originating IP addresses are almost always from the University of Minnesota’s network (128.101.x.x). No custom HTTP headers are used because the crawler does not use HTTP by design.

📊 Data Usage

Collected data—text files, directory structures, and menu entries—is used to maintain a publicly searchable index of all accessible gopherspace. This index is accessible via the gopher protocol at gopher://gopher.quux.org and is also mirrored on web gateways such as floodgap.com/gopher/gopher. The data serves academic research, digital preservation, and historical internet studies. The University of Minnesota provides the index for free and does not use it for commercial AI training or advertising.

⚙️ Rate Limiting Policy

Rate limiting of the Gopher crawler is rarely needed due to its extremely low request frequency (one every 3–5 seconds) and its restriction to port 70 on Gopher servers. However, if the crawler were to inadvertently connect to an HTTP server aggressively, administrators would rate-limit it based on its legacy nature and minimal impact; typical thresholds might block more than 5 requests per minute from the same IP.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.