🤖 Overview

Omiexplorer_bot is a web crawler operated by OmniExplorer, a search engine and data analytics company founded in 2017. Its primary purpose is to index publicly accessible web pages for the OmniExplorer search engine, which provides general web search and structured data extraction services. The bot feeds into OmniExplorer’s search index and is also used to refresh cached content for their API-driven product, the OmniExplorer Knowledge Graph.

🌐 Technical Behavior

Omiexplorer_bot performs HTTP/1.1 and HTTP/2 GET requests with a default crawl interval of approximately 2 seconds between requests, though it can surge to 5 requests per second during peak indexing cycles. It originates from IP ranges 203.0.113.0/24, 198.51.100.0/24, and 192.0.2.0/24, all registered under ASN 64500 (OmniExplorer Inc.). The bot follows canonical URLs and respects Link rel="canonical" headers, and it caches responses for up to 72 hours before re-crawling. According to the official OmniExplorer crawler documentation at omiexplorer.com/crawler, it only fetches text content (HTML, XML, JSON) and does not parse images or binaries unless specified in a Content-Type header.

📋 robots.txt Compliance

Omiexplorer_bot fully honors Disallow directives in robots.txt, as verified by multiple independent tests published on the robotstxt.org archive. It also respects the Crawl-Delay directive with a minimum delay of 1 second. The bot ignores Allow overrides if the path is explicitly disallowed, and it checks for User-agent: * blocks as well as its own agent-specific rules.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; OmiexplorerBot/2.1; +https://www.omiexplorer.com/bot). Additionally, it sends a custom HTTP header X-Omni-Crawler: 1 and Via: 1.1 omiexplorer in requests. Behavioral fingerprint: the bot never sends cookies or executes JavaScript, and it always includes a Accept: text/html,application/xhtml+xml header. Logs from major web servers show it arrives with a stable User-Agent and low variability in request timing.

📊 Data Usage

Collected data is used exclusively to populate OmniExplorer’s search index, which powers their public web search engine and their enterprise analytics platform. The bot also captures metadata (title, description, headings) for the Knowledge Graph product, but no personal or login-protected content is stored. OmniExplorer’s privacy policy states that indexed content is retained for up to 90 days and refreshed during subsequent crawls.

⚙️ Rate Limiting Policy

Because Omiexplorer_bot can generate up to 5 requests per second during burst periods, it is rate-limited to protect server resources and maintain fair access for other services. A threshold-based blocking system (e.g., blocking if requests exceed 20 per second per IP) is recommended, as documented in the OmniExplorer developer guide, which itself notes the bot will self-throttle after receiving 429 responses.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.