🤖 Overview

NoteworthyBot is a web crawler operated by Noteworthy AI, a company specializing in large-scale data extraction and AI model training. Officially documented at noteworthy.ai/bot, its primary purpose is to collect publicly accessible web content to feed into Noteworthy’s proprietary language models and analytics platforms. The bot was first publicly identified in early 2023 and has been observed actively crawling a wide range of domains for text, metadata, and structured data.

🌐 Technical Behavior

NoteworthyBot employs a headless Chromium-based browser engine to render JavaScript-dependent pages, allowing it to capture dynamic content often missed by simpler crawlers. According to the official documentation at noteworthy.ai/docs/crawler, it sends an average of 2–5 requests per second per source domain and respects the Crawl-Delay directive in robots.txt when present. The bot originates from a dynamic pool of IP addresses assigned to Amazon Web Services (us-east-1 and eu-west-1 regions) and Google Cloud Platform (us-central1), as verified by reverse DNS lookups and published IP ranges at noteworthy.ai/ips. It uses HTTP/1.1 with persistent connections and includes an Accept-Encoding: gzip header to reduce bandwidth consumption. The crawler follows a breadth-first strategy, typically starting from a domain’s sitemap.xml or homepage, and can respect If-Modified-Since headers to avoid re‑fetching unchanged resources. Noteworthy’s own GitHub repository at github.com/noteworthy-ai/crawler provides a public specification of its crawl policies, though the actual bot binary is proprietary.

📋 robots.txt Compliance

NoteworthyBot fully honors Disallow directives as documented in the official robots.txt policy page at noteworthy.ai/robots. It also respects Allow overrides and the Crawl-Delay value. Independent testing by the Web Robots Control Group (webrobots.org) in 2024 confirmed that the bot does not ignore exclusion rules under normal operation. However, the company notes that multiple simultaneous crawls may violate the intentional delay if not properly configured.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; NoteworthyBot/1.0; +https://noteworthy.ai/bot). Alternative strings may omit the Mozilla prefix. The bot sends a custom X-Bot-Identifier: noteworthybot header and a From: [email protected] header for contact. It does not impersonate browsers and includes a Referer field set to the page being crawled. Security researchers at Netcraft (netcraft.com) have published behavioral fingerprints noting the bot’s consistent request timing and lack of mouse‑movement simulation.

📊 Data Usage

Collected web content is used to train Noteworthy’s N‑series language models and to enhance their Semantic Search Engine, both detailed at noteworthy.ai/products. The data is also aggregated into anonymized analytics reports sold to enterprise clients under the WebInsight brand. Noteworthy explicitly states that no personal or copyrighted material is retained beyond the raw extraction required for model training, per their privacy policy at noteworthy.ai/privacy.

⚙️ Rate Limiting Policy

NoteworthyBot is rate‑limited because its dynamic IP pool and headless rendering can generate bursts that mimic distributed denial‑of‑service activity if left unchecked. The recommended threshold is 10 requests per second per IP with a 60‑second ban after exceeding 50 requests in a rolling window, as advised by Noteworthy’s own operator guidelines at noteworthy.ai/rate-limit. This policy balances the bot’s legitimate data collection needs with site stability for other traffic.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.