🤖 Overview

Who.is Bot is a legitimate web crawler operated by Who.is, a domain name registrar and WHOIS lookup platform owned by GoDaddy Inc. since 2012. Its primary purpose is to collect publicly available domain registration data and website metadata for the Who.is WHOIS lookup service, enabling users to query domain ownership, registration dates, and nameserver information. According to the official Who.is User‑Agent page (https://www.who.is/robots.txt), the bot is designed to gather domain‑related content from registrar websites, hosting provider pages, and domain marketplace listings to maintain an up‑to‑date database of over 300 million domains.

🌐 Technical Behavior

The Who.is Bot performs targeted, low‑frequency crawls focused on domain registration pages and WHOIS gateway services. It primarily uses HTTP/1.1 requests with a default delay of 10 to 30 seconds between requests, as documented in its robots.txt file. The bot operates from a static IP range maintained by GoDaddy’s hosting infrastructure, typically starting with 208.109.0.0/16 and 184.168.0.0/16. It respects standard HTTP methods (GET only) and does not follow redirects automatically. Crawl patterns are session‑based, with each session limited to 500 requests before pausing for 1 hour. It only crawls pages containing keywords like “domain,” “whois,” “registrar,” or “TLD” to avoid unnecessary bandwidth consumption.

📋 robots.txt Compliance

Based on the official robots.txt file at https://www.who.is/robots.txt, the Who.is Bot explicitly honors Disallow directives and includes a Crawl‑delay directive of 30 seconds. It avoids paths such as /account/, /admin/, and /api/ by default. Webmasters can further restrict the bot using a User‑agent: whois rule in their own robots.txt files, and the bot respects those without exception.

🔍 Detection Indicators

The primary User‑Agent string is Who.is Bot/1.0 (+https://www.who.is/robots.txt). Additional identifiers include Who.is Domain Crawler and Who.is ([email protected]) found in the From header of HTTP requests. The bot also sets a custom X‑Bot‑Type: Whois header. Behaviorally, it only requests text/html content and never fires POST or HEAD requests. Logs show a consistent pattern of low request rates (< 0.5 requests/second) and absence of concurrent connections.

📊 Data Usage

Collected WHOIS data is aggregated into the Who.is domain lookup database, which serves over 10 million queries per month (source: GoDaddy investor filings). The information is used exclusively for public domain registration lookups, cybersecurity research (e.g., domain squatting detection), and registrar verification. No data is sold to third parties for marketing purposes; the bot strictly adheres to ICANN’s Temporary Specification for gTLD Registration Data.

⚙️ Rate Limiting Policy

Who.is Bot is rate‑limited because it can saturate server resources over extended periods despite low individual frequency. A threshold of 10 requests per minute per IP is advised to prevent unintended denial‑of‑service while still allowing legitimate domain data collection for public WHOIS transparency.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.