🤖 Overview

Nextthing is a web crawler operated by Nextthing.org, a search engine and AI research platform founded in 2023. Its primary purpose is to collect publicly accessible web content to train large language models (LLMs) and improve the nextthing.org search index. According to their official documentation at https://nextthing.org/crawler, the bot is described as a “responsible, transparent crawler” that respects robot exclusion protocols and provides a verification endpoint.

🌐 Technical Behavior

The crawler, identified by the User-Agent string Nextthing/1.0, issues HTTP/1.1 GET requests at a default rate of approximately 10 requests per second, with configurable delays via a Crawl-Delay directive. It supports both IPv4 and IPv6 and publishes its IP address ranges via a dedicated DNS lookup (e.g., _nextthing._tcp.crawler.nextthing.org). The bot only fetches HTML, CSS, JavaScript, and common text-based file types; it explicitly avoids binary files (images, videos, archives) unless linked as primary content. It sets a custom X-Robots-Tag header during requests to allow site owners to block specific pages without modifying robots.txt.

📋 robots.txt Compliance

Nextthing explicitly states in its documentation that it honors all Disallow directives in robots.txt and respects the Crawl-Delay value if provided. It also supports the X-Robots-Tag HTTP header for per-URL blocking. The crawler team provides a report blocked URL form for site owners who cannot use robots.txt.

🔍 Detection Indicators

The definitive User-Agent string is Nextthing/1.0, but variations like Mozilla/5.0 (compatible; Nextthing/1.0; +https://nextthing.org/crawler) have been observed. A behavioral fingerprint is the X-Forwarded-For header containing a predictable pattern of their known IP ranges. The bot also includes a Referer header of https://nextthing.org and a User-Agent token with a forward-slash version number.

📊 Data Usage

Collected data is used for two primary purposes: training Nextthing’s AI models (including their proprietary large language model) and populating the nextthing.org search index for public use. The company publishes a list of corpus sources and allows opt-out requests via their privacy page.

⚙️ Rate Limiting Policy

Despite its legitimacy, Nextthing is rate-limited because its default crawl rate (10 req/s) can overload small or medium-sized web servers. The recommended policy is to set a rate limit of 5 requests per second per IP, with threshold-based blocking triggered after sustained bursts exceeding 15 req/s for 30 seconds. This ensures fair resource allocation without entirely blocking the crawler.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.