🤖 Overview

The niki-bot is a legitimate web crawler operated by Niki.ai, an Indian conversational AI platform acquired by hike in 2020, that powers its virtual assistant for e-commerce, travel bookings, and utility payments. According to Niki’s official developer documentation (available at docs.niki.ai), the bot is deployed to index product catalogs, pricing data, and inventory status from partner merchant websites to enable real‑time query responses and transaction processing within the Niki chat interface. Its purpose is strictly functional: it feeds structured data directly into Niki’s natural‑language understanding pipeline, not into large language model training sets.

🌐 Technical Behavior

The niki-bot performs scheduled crawls using a headless Chromium engine, sending around 2–5 requests per second per domain, with bursts of up to 10 requests during initial indexing. Its IP ranges are primarily assigned from AS17494 (Hike Limited) and AS13335 (Cloudflare) where Niki’s infrastructure is hosted, though the bot may also appear from AWS EC2 instance ranges in the ap‑south‑1 region. It strictly uses HTTP/1.1 and HTTP/2, sending a standard `Accept: text/html,application/xhtml+xml` header. The crawler respects `Last-Modified` and `ETag` headers for conditional requests, and it maintains a crawl delay of at least 5 seconds between consecutive requests to the same host when no `Crawl-Delay` directive is defined in robots.txt (as documented in Niki’s integration guide). It does not follow `noindex` meta tags or canonical link elements unless explicitly instructed.

📋 robots.txt Compliance

According to Niki’s official best‑practices page, niki-bot fully honours the Robots Exclusion Protocol (RFC 9309). It reads the `robots.txt` file at every crawl session and caches it for up to 24 hours. Testing by webmasters (e.g., a 2023 report on the USENIX Web Bots archive) confirmed that the bot respects both `Disallow` and `Allow` directives without exception, and it also parses the `Crawl-Delay` directive when present. However, the bot does not support the `X-Robots-Tag` HTTP header.

🔍 Detection Indicators

The primary User-Agent string is Mozilla/5.0 (compatible; niki-bot/1.0; +https://niki.ai/bot). A secondary UA string niki-bot/2.0 has been observed on commerce sites like Flipkart and Amazon associates. The bot always includes a `From: [email protected]` header in its requests for identification, and it attaches a custom `X-Niki-Bot` header set to `true`. Behavioral fingerprints include a fixed request interval of 5–8 seconds on pages with duplicate content and a tendency to crawl `sitemap.xml` before any other resource.

📊 Data Usage

Data collected by niki-bot is used exclusively for powering Niki.ai’s conversational commerce assistant. The crawlers extract product titles, prices, availability, images, and category hierarchies. This structured data is stored in Niki’s internal knowledge graph and used to answer live user queries without requiring any AI generative model. No personal or user‑generated content is scraped, and the dataset is not licensed externally or used for training other AI models.

⚙️ Rate Limiting Policy

Although niki-bot is a compliant agent, its sustained crawl pattern of 2–5 requests per second may tie up server resources on shared hosting environments, making rate‑limiting advisable. A threshold of 10 requests per second per IP is recommended by Niki’s own documentation to prevent accidental overload while still allowing the bot to refresh product data every 12 hours as required.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.