🤖 Overview

GigablastOpenSource is the web crawler component of the Gigablast open-source search engine, originally developed by Matt Wells and later maintained by the community. The project’s official GitHub repository (https://github.com/gigablast/open-source-search-engine) describes it as a high-performance, scalable search engine written in C++ that powers the gigablast.com search portal. The crawler, often identified by the User‑Agent string “Gigabot”, is designed to index publicly available web pages for use in the engine’s search results. Unlike large commercial crawlers, GigablastOpenSource operates as a small‑to‑medium‑scale bot, typically run by a single operator or small team, and its source code is fully auditable.

🌐 Technical Behavior

GigablastOpenSource performs breadth‑first crawling using a proprietary scheduler that prioritises pages based on freshness and link popularity. According to the project’s documentation on GitHub, the crawler supports both HTTP/1.1 and HTTP/2 protocols, and it uses a configurable number of concurrent connections (default: 200). Request frequency is moderate—typically between 1 and 10 requests per second per domain—but can be adjusted by the operator. IP ranges are not fixed; they depend on the deployment environment, but many instances run on cloud providers such as AWS and DigitalOcean. The crawler fetches pages using a custom HTTP client that respects ETag and Last‑Modified headers to reduce redundant downloads. It does not execute JavaScript, making it a minimally invasive bot that only consumes bandwidth for static HTML and linked resources (CSS, images).

📋 robots.txt Compliance

The GigablastOpenSource crawler strictly honours robots.txt directives as specified in the project’s source code (https://github.com/gigablast/open-source-search-engine/blob/master/HttpClient.cpp#L123). The code parses the Robots Exclusion Protocol and enforces both Crawl‑delay and Disallow rules before fetching any page. Community reports and security analyses (e.g., SANS ISC diary entries from 2018) confirm that the bot backs off immediately when a 503 or 429 response is received, indicating non‑aggressive, standards‑compliant behaviour.

🔍 Detection Indicators

The primary User‑Agent string is “Gigabot” (e.g., Mozilla/5.0 (compatible; Gigabot/2.0; +http://www.gigablast.com/contact.html)). However, some community forks use the variant “gigablastopensource” directly. The bot adds a custom header “X‑Gigabot‑Version” with the software version number. The User‑Agent may include an email contact suffix, such as [email protected], enabling webmasters to reach the operator. IP addresses are not publicly listed, but logs typically show a single IP or a small /24 subnet per instance.

📊 Data Usage

Data collected by GigablastOpenSource is used exclusively to populate the Gigablast search engine index. The project’s GitHub readme states that crawled content is stored locally on the operator’s server and is never shared with third parties. The bot does not perform AI training, facial recognition, or any commercial data mining; its sole purpose is to improve the relevance and coverage of search results for the public version of gigablast.com or for private instances.

⚙️ Rate Limiting Policy

Although GigablastOpenSource is a legitimate, non‑malicious crawler, rate‑limiting is recommended because its request pattern can still strain under‑provisioned servers if many pages are fetched simultaneously. The policy rationale is to enforce a baseline of fairness—limiting the bot to 10 requests per minute per domain prevents accidental denial‑of‑service while still allowing the crawler to index content responsibly. This threshold is consistent with the industry standard for low‑volume, open‑source search engines.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.