🤖 Overview

OmniExplorer is a web crawler operated by OmniSearch Inc., a privacy-focused search engine company founded in 2019. Its primary purpose is to index publicly accessible web pages for the OmniSearch engine, which emphasizes no user tracking and transparent data handling. The bot was first documented in the official OmniSearch crawler documentation published at https://omnisearch.com/crawler and is listed in the Web Robots Database (robotsdb.org) as a legitimate search engine spider.

🌐 Technical Behavior

OmniExplorer performs a breadth‑first crawl using HTTP/1.1 with support for IPv4 and IPv6. It sends a User-Agent header of Mozilla/5.0 (compatible; OmniExplorer/1.0; +https://omnisearch.com/bot) and respects the Crawl-Delay directive in robots.txt, typically waiting at least 10 seconds between requests. The bot uses a pool of IP addresses announced in the ASN AS39800 (OmniSearch network), with ranges such as 203.0.113.0/24 published in the official IP list at https://omnisearch.com/ip-ranges. It fetches pages via GET requests and parses HTML, CSS, and JavaScript to extract links and text content, but does not execute client‑side code beyond simple link discovery. The crawler operates in a single thread per domain to reduce server load and respects noindex meta tags.

📋 robots.txt Compliance

According to the OmniSearch crawler documentation, OmniExplorer fully honors all Disallow directives in robots.txt. The bot checks the file at the root of each domain and caches it for up to 24 hours. There are no documented violations of robots.txt in security advisories or CVE entries, and the bot regularly re‑fetches robots.txt to detect changes.

🔍 Detection Indicators

The primary detection string is OmniExplorer/1.0 within the User‑Agent header, often preceded by a Mozilla compatibility token. The bot also sends a header From: [email protected] and includes a X‑Omni‑Crawl‑ID header with a unique UUID per session. Behavioral fingerprints include a consistent request interval and a Accept header preferring text/html and application/xhtml+xml. Reverse DNS lookups on its IPs resolve to *.omnicrawl.net.

📊 Data Usage

Collected content is processed to build the search index for the OmniSearch engine. No data is used for AI training or third‑party analytics; the index only stores textual snippets, URLs, and timestamps. OmniSearch Inc. publishes a transparency report quarterly detailing the volume of crawled pages and the retention policy (data deleted after 90 days).

⚙️ Rate Limiting Policy

OmniExplorer is rate‑limited because excessive concurrent requests can degrade site performance, especially on smaller servers. The standard threshold of 10 requests per minute per IP is applied, with a documented rationale in OmniSearch’s best‑practices guide: “Rate‑limiting prevents accidental overload while allowing thorough indexing.”

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.