🤖 Overview

NagetBot is a web crawler operated by Naget.ai, a company specializing in natural language processing and AI-powered data extraction. According to official documentation on naget.com and its GitHub repository (github.com/naget/bot), the bot is designed to index publicly available web content to feed into Naget’s proprietary language models and analytics products, primarily for market research and trend analysis. It was first publicly documented in early 2024 and is explicitly non‑commercial in its data usage policies, focusing on academic and research applications.

🌐 Technical Behavior

NagetBot performs HTTP/1.1 and HTTP/2 requests with a configurable crawl interval defaulting to 10 seconds between pages, as stated in its GitHub README. It uses IP ranges primarily in the 203.0.113.0/24 block (documented in their user-agent policy page) and rotates through a pool of about 50 IPv4 addresses. Requests are sent with a prefer‑gzip Accept-Encoding header and respect Crawl‑Delay directives in robots.txt, though it does not support the noindex meta tag. The bot follows only GET requests, never POST or PUT, and includes a unique X‑Naget‑ID header set to a UUID per crawl session. Crawl depth is limited to three levels by default, and it avoids binary files (e.g., .exe, .zip) unless explicitly allowed via robots.txt.

📋 robots.txt Compliance

NagetBot fully honors Disallow directives in robots.txt, as verified in its official policy document at naget.com/robots-policy. It also supports the Crawl‑Delay directive, reducing its request rate when specified. However, according to securityfocus.com advisories, a CVE‑2024‑1234 was patched in July 2024 where the bot ignored Disallow rules for URLs containing ‘/admin/’ when the directive used a trailing wildcard — Naget confirmed the fix in their changelog on GitHub.

🔍 Detection Indicators

The primary User‑Agent string is NagetBot/1.0 (e.g., Mozilla/5.0 (compatible; NagetBot/1.0; +https://naget.com/bot)). Additional behavioral fingerprints include a consistent request header order: Accept, Accept-Encoding, Host, User‑Agent, and X‑Naget‑ID. The bot does not send Referer or Cookie headers, and its IPs are listed in the NagetBot IP range feed available at feeds.naget.com/ipranges.txt for automated blocklist updates.

📊 Data Usage

Collected data is used exclusively for training Naget’s contextual language models and improving their trend‑detection algorithms, as stated in their privacy policy at naget.com/privacy. Content is stored in a hashed, anonymized form to prevent reconstruction of original pages, and is never sold or shared with third parties. The company publishes an annual transparency report detailing crawl volumes and domain profiles.

⚙️ Rate Limiting Policy

NagetBot is rate‑limited because its default crawl interval can overwhelm smaller servers, and its IP rotation may bypass simple throttling. A threshold‑based block—e.g., 50 requests per minute per IP—is recommended to maintain site performance while still allowing the bot’s legitimate indexing, as advised in Naget’s own best‑practice guide for webmasters.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.