Skip to main content

Boteraser | Website and Server Security Solutions

G-i-g-a-b-o-t

Bot User-Agent: g-i-g-a-b-o-t

🤖 Overview

Gigabot is the official web crawler for the Gigablast open-source search engine, originally developed by Matt Wells in 2000. Its purpose is to discover and index publicly accessible web pages, contributing to an independent search index available at gigablast.com. The full source code, including the crawler, is maintained on GitHub under the repository github.com/gigablast/open-source-search-engine, with ongoing updates.

🌐 Technical Behavior

Gigabot employs a breadth-first crawling algorithm, starting from a seed list and following hyperlinks within configurable scope. It fetches pages over HTTP and HTTPS, parsing HTML, plain text, PDF, and XML content types. Default request intervals are set to a few seconds per host, but operators can adjust concurrency and delay via command-line flags. The crawler does not use a fixed IP range; it runs on servers owned by the operator, which may vary. Each request includes the User-Agent "Gigabot/1.0" (full string: Mozilla/5.0 (compatible; Gigabot/1.0; +http://www.gigablast.com/gigabot.html)). It also sends an Accept-Encoding header for gzip compression, as documented in the GitHub source. The crawler is implemented in C++ for performance.

📋 robots.txt Compliance

According to the official policy page at gigablast.com/gigabot.html, Gigabot strictly honors robots.txt directives, including Disallow and Crawl-Delay settings. The implementation is verified in the source code (gigablast/open-source-search-engine/blob/master/robotstxt.cpp), which parses robot exclusion rules before each request. It also respects X-Robots-Tag HTTP headers. Webmaster community

Free Traffic Analysis

What's Actually Crawling Your Website?

Discover which unwanted bots are being blocked on your site, how often they hit, and where they come from — real data from your own traffic, not guesswork.

🔍 Scan My Site Free

Powered by JA4 fingerprinting, honeypot traps & behavioral analysis

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.

✓