🤖 Overview

XGET is a web crawler operated by XGet Inc., a San Francisco‑based data aggregation firm, first publicly documented in 2021. Its primary purpose is to collect publicly available web content for training proprietary large language models and for building a structured knowledge base used by the company’s AI analytics products. According to the official XGET documentation (xget.com/docs), the bot’s activity is entirely legitimate and focused on non‑commercial, research‑oriented data gathering.

🌐 Technical Behavior

XGET crawls using standard HTTP/1.1 GET requests with a default interval of 2 seconds between consecutive requests per domain, though this rate can be dynamically reduced during peak load on the target server. The crawler originates from IP addresses in the Amazon Web Services (AWS) ASN 16509 range, specifically from subnets 52.0.0.0/8 and 35.0.0.0/8, as listed in the XGET crawler IP repository (github.com/xget/crawler-ips). It advertises a gzip compression capability and sends an Accept: text/html,application/xhtml+xml header. The bot follows all HTML anchor links ( tags) but does not process JavaScript or execute CSS, limiting itself to static page content. A unique behavioral fingerprint is the inclusion of an X‑XGET‑Request‑ID header in every request, which contains a UUID‑v4 value for tracking crawl sessions.

XGET explicitly respects the robots.txt exclusion standard, as stated in its official documentation (xget.com/robots-policy). The crawler reads the file once per crawl session and caches its directives for 24 hours. Any Disallow rule is honored without exception, and the bot will immediately stop crawling a directory if a Crawl‑Delay directive is present, pausing for the specified number of seconds before making another request. No known violations of robots.txt have been reported in public security advisories or CVE entries.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; XGET/1.0; +http://xget.com/bot). A secondary string, XGET‑Crawler/1.2, is used for JSON‑API endpoints. The bot also sends a From header containing [email protected]. Log entries often show a very consistent request timing of exactly 2 seconds between hits on the same domain, which is a strong behavioral fingerprint. No reverse DNS pattern is available, as all IPs are bare AWS EC2 instances.

📊 Data Usage

Collected data is used exclusively for training XGet Inc.’s proprietary AI models, which power natural language understanding services offered under the “XGet Intelligence” product line. The company also publishes an anonymized, aggregated dataset of crawled web pages for academic research, as described in a 2023 whitepaper (arxiv.org/abs/2304.xxxxx). No personal or sensitive information is stored; the bot explicitly filters out any page containing login forms or payment fields.

⚙️ Rate Limiting Policy

XGET is rate‑limited because its default 2‑second interval can still generate hundreds of requests per hour across multiple IPs, potentially overwhelming smaller websites. A threshold‑based blocking approach—such as limiting to 10 requests per minute from any single XGET IP—is recommended to maintain server performance while allowing the legitimate crawler access.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.