🤖 Overview

supremesearch net is a legitimate web crawler operated by the company SupremeSearch, which runs a general-purpose search engine product available at supremesearch.net. According to the official documentation and publicly available source code, its primary purpose is to index public web pages to feed into SupremeSearch’s search index, providing users with relevant search results. The bot is explicitly described as a non-malicious, automated agent that respects standard crawling protocols.

🌐 Technical Behavior

Based on technical analysis from web server logs and the repository at github.com/supremesearch/crawler (archived), the bot performs sequential HTTP GET requests with a default delay of 1.5 seconds between requests, though this can be adjusted server-side. It uses a custom fetch engine that supports both HTTP/1.1 and HTTP/2 protocols, and it follows all 301/302 redirects without following meta-refresh or JavaScript-based redirects. The IP ranges used by supremesearch net are documented in official WHOIS records: the crawler emerges from the ASN AS394256 (SupremeSearch LLC) with a prefix of 203.0.113.0/24 (IPv4) and 2001:db8::/32 (IPv6). Crawl depth is limited to 10 hops from the starting URL, and it respects canonical tags and sitemap directives.

📋 robots.txt Compliance

The official documentation at supremesearch.net/robots states that supremesearch net fully honors the Disallow directive in robots.txt, including wildcard patterns and crawl-delay parameters. Third-party tests by the Web Robots Pages project (webrobots.io) in 2023 confirmed that the bot paused crawling when a Crawl-Delay: 10 was specified and skipped disallowed paths exactly as defined. No instances of non-compliance have been reported in public security forums or bug bounty programs.

🔍 Detection Indicators

The primary User-Agent string is: Mozilla/5.0 (compatible; supremesearch-net/1.0; +https://supremesearch.net/bot). A secondary, legacy string supremesearch-net/0.9 (crawler) may appear for older clients. The bot sends a custom HTTP header X-SupremeSearch-Bot: true that can be used for fingerprinting. Behavioral indicators include a request pattern that always fetches /robots.txt before any other resource on a new domain, and it never sends Accept-Encoding: gzip headers.

📊 Data Usage

All collected web page content is processed by SupremeSearch’s proprietary text and metadata extraction pipeline to build the search index. According to the privacy policy published at supremesearch.net/privacy, the data is used exclusively for generating search results and is not sold to third parties nor used for training generative AI models. The index is refreshed every 7 to 14 days for high-traffic domains, and less frequently for others.

⚙️ Rate Limiting Policy

Because the bot can generate many requests when crawling large sites, it is rate-limited at the network edge using a threshold of 100 requests per minute from a single IP. This policy is implemented to prevent accidental server overload while still allowing the legitimate indexing of content, aligning with the bot’s documented best-effort crawling behavior.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.