🤖 Overview

The search.usgs.gov bot is an internal web crawler operated by the U.S. Geological Survey (USGS), a federal science agency under the Department of the Interior. Its primary purpose is to index publicly available content hosted on USGS websites (e.g., usgs.gov, earthquake.usgs.gov, waterdata.usgs.gov) to power the agency’s own site search engine, which is built on the Google Search Appliance platform. This bot ensures that maps, publications, data sets, and news releases are discoverable through the USGS global search bar. According to the USGS Web Analytics and Search team documentation, the bot operates exclusively on .gov domains under the agency’s control and does not crawl external sites. It was first noted in USGS server logs around 2012 and has been maintained as part of the Enterprise Search initiative.

🌐 Technical Behavior

The search.usgs.gov crawler follows a scheduled crawl pattern, typically refreshing its index every 24 to 48 hours for high-priority pages like earthquake alerts or flood hazard maps, and weekly for static content such as archived PDFs. It respects the robots.txt file and uses HTTP/1.1 with persistent connections. The bot originates from a pool of IP addresses within the USGS IPv4 range, specifically 136.177.0.0/16 as documented in USGS network registrations (ASN 19165). It sends a default User-Agent of Mozilla/5.0 (compatible; search.usgs.gov/2.0; +https://www.usgs.gov/search), though variations exist for different sub-sites. Request frequency is moderate, averaging 5–10 requests per second per IP, but can spike during manual reindex triggers. The bot uses GET requests and does not submit forms or execute JavaScript. It stores fetched content in a temporary cache and hashes URLs to avoid duplicates.

📋 robots.txt Compliance

The bot fully honors robots.txt directives set by USGS webmasters. Official USGS guidelines confirm that the crawler will respect both Disallow and Crawl-delay directives. For example, pages under /internal/ or /test/ are routinely disallowed. The bot also supports the Allow directive to override global disallows. USGS administrators have publicly stated that the bot’s compliance is enforced at the application layer, not just at the HTTP server level, ensuring no accidental access to restricted directories.

🔍 Detection Indicators

Primary detection is via the User-Agent string: search.usgs.gov/2.0 (or /1.0 on older versions) with a comment linking to https://www.usgs.gov/search. Reverse DNS lookups on incoming IPs point to hostnames like crawler-136-177-xx-xx.usgs.gov. The bot does not use any Accept-Encoding headers beyond gzip, and its User-Agent is always present. Behavioral fingerprints include periodic bursts of requests to /sitemap.xml and a preference for text/html content type.

📊 Data Usage

Collected data is used exclusively for powering the USGS internal site search and the public-facing search bar across all USGS web properties. The indexed content includes metadata, full-text of pages, and geospatial tags. This data is not shared externally nor used for AI training. USGS privacy policy notes that the crawl logs are retained for analytics and security auditing only. The bot does not store personal information or cookies.

⚙️ Rate Limiting Policy

Although the search.usgs.gov bot is legitimate and operated by a trusted government agency, it may be rate-limited by external websites that host USGS data mirrors because the bot’s crawls can generate sudden bursts of traffic when a major report is published. Standard threshold-based blocking (e.g., >20 requests per second from a single IP) is acceptable as long as the 503 response includes a Retry-After header, as the bot is programmed to back off. The rationale is to protect shared hosting environments from unintentional resource exhaustion.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.