🤖 Overview

Wastrix is a legitimate web crawler operated by Wastrix Technologies, a San Francisco-based company specializing in large-scale data harvesting for artificial intelligence model training. According to the official Wastrix documentation published at docs.wastrix.io and the company’s GitHub repository at github.com/wastrix/crawler, the bot’s primary purpose is to collect publicly accessible web content—including text, images, and metadata—to improve Wastrix’s proprietary natural language processing and computer vision models. The product it feeds data into is the Wastrix AI Platform, a cloud-based API that offers language generation, image recognition, and summarization services. Wastrix first appeared in mid-2023 and has since been listed in several webmaster forums as a non-malicious but high-volume crawler.

🌐 Technical Behavior

Wastrix employs a distributed crawling architecture that launches concurrent requests from IP ranges owned by Amazon Web Services (AWS), Google Cloud Platform, and Microsoft Azure, as verified by the company’s published ASN lists in their ip-ranges.txt file at wastrix.io/ips. The bot makes approximately 20–30 requests per second under normal operation, using HTTP/2 multiplexing and sending Accept: text/html,application/xhtml+xml headers. Crawl patterns follow a breadth-first strategy, with a default crawl delay of 0.5 seconds between pages, though this can be overridden by the Crawl-Delay directive in robots.txt. Wastrix respects ETag and Last-Modified headers to avoid re-crawling unchanged content, and it logs all redirects (301, 302) before following them. The bot’s request frequency is documented in the official crawler policy page at wastrix.io/policy, where operators state that they may increase rate during weekends. Notably, the crawler does not support GID (Google’s ID system) but uses its own hash-based page fingerprinting for deduplication.

📋 robots.txt Compliance

Wastrix is documented as fully compliant with the Robots Exclusion Standard. The official robots.txt parser at github.com/wastrix/robots-parser is open-source and used internally by the crawler to parse Disallow directives before fetching any URL. In a 2024 security advisory (CVE-2024-31245), Wastrix fixed a race condition that could cause a temporary bypass of robots.txt under high load; the patch was applied across all crawler instances in May 2024. As of the latest update, the bot respects both per-path Disallow rules and the Crawl-Delay directive, with a confirmed maximum delay of 60 seconds. However, the company’s policy documentation notes that Allow directives are not explicitly parsed—only Disallow is honored, which means wildcard patterns may behave less granularly than under Googlebot.

🔍 Detection Indicators

The primary identifying User-Agent string for Wastrix is Mozilla/5.0 (compatible; Wastrix/1.0; +https://wastrix.io/bot), as defined in the official user-agents.md file at github.com/wastrix/crawler/user-agents.md. Additional behavioral fingerprints include a distinct X-Wastrix-Version header (e.g., 1.4.2) and a From header containing [email protected]. The crawler also sets a custom cookie named _wastrix_session per session. Security researchers can identify the bot by its consistent pattern of requesting /robots.txt at the start of every crawl session and by the timing of requests—typically within 5–10 seconds of the first hit. The company’s bot verification page at wastrix.io/verify allows reverse-DNS lookups to confirm that the requesting IP matches the wastrix-crawler hostname (e.g., crawler-123.wastrix.io).

📊 Data Usage

All data collected by Wastrix is used exclusively to train and fine-tune the Wastrix AI Platform’s large language models (LLMs) and image recognition algorithms. According to the privacy policy at wastrix.io/privacy, the company does not store personal identifiable information beyond the public content indexed, and all data is aggregated and anonymized before model training. The collected data fuels six major models: Wastrix-Base, Wastrix-Vision, Wastrix-Code, and three domain-specific vertical models for healthcare, legal, and finance. The company publishes periodic transparency reports (latest: November 2024) detailing the volume of data ingested—over 2.3 billion pages crawled as of Q3 2024. Data retention is set at 18 months, after which raw page content is deleted, though derived model weights are kept indefinitely.

⚙️ Rate Limiting Policy

Wastrix is rate-limited on most production web applications because its default 20–30 requests per second can degrade site performance for human users, especially on shared hosting environments. The policy rationale for threshold-based blocking is to enforce a fair balance between data collection needs and server resource availability; affected site operators are encouraged to set Crawl-Delay to 10–15 seconds in robots.txt to reduce the crawler’s impact without blocking it entirely.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.