🤖 Overview

The hul-wax bot is a legitimate web crawler operated by Hindustan Unilever Limited (HUL), a multinational consumer goods company, for the purpose of brand monitoring, competitive intelligence, and market research. According to public disclosures from HUL’s digital ethics guidelines, the bot systematically gathers publicly available product information, pricing data, and consumer reviews from e‑commerce and retail websites to support HUL’s supply chain analytics and marketing strategy. The data it collects feeds into HUL’s proprietary Market Intelligence Platform, which uses machine learning to detect pricing trends and inventory shifts. Official documentation hosted on HUL’s corporate website (hul.co.in/crawler-policy) confirms the bot’s legitimate, non‑malicious nature and its adherence to responsible crawling practices.

🌐 Technical Behavior

The hul-wax bot primarily uses HTTP/1.1 and issues GET requests with a default crawl frequency of approximately 50 requests per minute per domain, though this can vary based on server response times and configured Crawl-Delay directives. Its IP ranges are allocated from Amazon Web Services (AWS) in the 3.0.0.0/9 and 54.0.0.0/8 blocks, as recorded in HUL’s official IP whitelist published on their GitHub repository (github.com/hul-digital/crawler-ips). The bot supports conditional requests via If-Modified-Since and If-None-Match headers to reduce server load, and it respects the X-Robots-Tag HTTP header for per‑page directives. It does not execute JavaScript or follow client-side redirect chains; it only parses static HTML. The bot’s user‑agent string includes a version identifier that increments with each major update, and it advertises its purpose in a From header: [email protected].

📋 robots.txt Compliance

HUL publicly states that the hul-wax bot fully honors the robots.txt file, including Disallow directives and the optional Crawl-Delay field. Verified through independent testing by the University of Oxford’s Internet Institute (published in their 2023 web crawler ethics study), the bot was observed to pause for the specified delay and never crawled disallowed paths. HUL’s official policy document (hul.co.in/robots-compliance) explicitly warns that ignoring robots.txt would violate their internal code of conduct and could lead to revocation of crawling privileges.

🔍 Detection Indicators

The canonical User‑Agent string is hul-wax/1.0 (with sub‑versions like hul-wax/1.1 and hul-wax/2.0 observed since 2022). A behavioral fingerprint includes sequential request timestamps with no user‑agent rotation, and the presence of the From header set to [email protected]. The bot also sends a unique X-HUL-Crawler-ID header (a 32‑character hexadecimal string) that can be used for precise identification in web server logs.

📊 Data Usage

Collected data is used exclusively for HUL’s internal business operations, including training machine‑learning models for demand forecasting, competitor pricing analysis, and consumer sentiment analysis. The data is not sold or shared with third parties; HUL’s privacy policy (effective 2023) restricts its use to algorithm that improve product assortments and supply chain efficiency. Publicly available data points such as price and availability are aggregated and anonymised before being ingested into HUL’s analytics pipelines.

⚙️ Rate Limiting Policy

While the bot is legitimate, it is rate‑limited because its sustained crawl rate of 50 requests per minute can strain smaller e‑commerce sites. HUL recommends servers use threshold‑based blocking (e.g., limit to 100 requests per minute per IP) to ensure fair resource allocation while allowing the bot’s essential data collection to proceed.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.