🤖 Overview

gaisbot is a web crawler operated by Global AI Systems (GAIS), a company specializing in large-scale data acquisition for artificial intelligence training. First documented in early 2024, the bot systematically collects publicly accessible web content to feed into GAIS’s proprietary language model training pipelines. According to official documentation published at gais.ai/crawling-policy, the bot is designed to support the development of multilingual and domain-specific AI models, focusing on high-quality textual data from diverse sources.

🌐 Technical Behavior

gaisbot performs distributed crawling using a fleet of servers hosted across Amazon Web Services (AWS) and Google Cloud Platform (GCP), with IP ranges publicly listed in the gaisbot-ips.txt file at gais.ai/robots.txt. The crawler issues requests at a default rate of one request per 2 seconds per IP, but can burst to higher frequencies during initial indexation of new domains. It uses HTTP/1.1 with keep-alive connections and respects ETag and Last-Modified headers to avoid redundant downloads. The bot prioritizes content freshness, re-crawling pages with a Cache-Control: max-age=3600 header every 24 hours, while static resources (e.g., PDFs, CSVs) are fetched only once unless updated. Crawling occurs primarily during non-peak hours in the target server’s timezone, as specified in GAIS’s operational guidelines.

📋 robots.txt Compliance

gaisbot fully honors Disallow directives in robots.txt, as verified in independent tests by the Robotstxt.com compliance checker. GAIS explicitly states in its Crawler Policy (accessed at gais.ai/policy) that the bot will not crawl any path marked with Disallow, including nested subdirectories. Additionally, it supports the Crawl-Delay directive, allowing webmasters to set a minimum interval between successive requests. No evidence of robots.txt bypass has been reported in public security advisories or CVE entries as of late 2024.

🔍 Detection Indicators

The primary User-Agent string is gaisbot/1.0 (+https://gais.ai/crawler), with a secondary variant gaisbot/2.0 (compatible; gaisbot/2.0; +https://gais.ai/crawler-bot) used for experimental crawling. Behavioral fingerprints include a consistent From header containing [email protected], and a X-GAIS-Crawler header set to true. The bot also sends a Accept-Language: en-US,en;q=0.9 header and a Accept-Encoding: gzip, deflate header typical of modern crawlers.

📊 Data Usage

Collected data is used exclusively for training GAIS’s large language models (such as the GAIS-LLM series), improving natural language understanding, translation, and content generation capabilities. GAIS does not index content for public search engines or sell the raw data to third parties; instead, it aggregates cleaned and annotated datasets that are later used in model fine-tuning and evaluation benchmarks. The company’s privacy policy affirms that no personally identifiable information (PII) is intentionally harvested, and automated filters strip known PII patterns before storage.

⚙️ Rate Limiting Policy

gaisbot is rate-limited because its distributed crawling, while compliant, can still consume significant server resources when scanning large sites or during initial deep crawls. The policy rationale for threshold-based blocking is to prevent accidental service degradation while allowing legitimate data collection; most webmasters set a limit of 10 requests per second per IP with a 429 response after exceeding the cap. GAIS recommends a burst of 20 requests before blocking, as documented in their webmaster FAQ at gais.ai/rate-limits.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.