🤖 Overview

Demon is a web crawler operated by Demon Internet, a UK-based internet service provider founded in 1992, originally used to build and maintain the Demon Search portal for indexing publicly accessible web content. Historically, it was one of the earliest commercial search engines in the UK, serving both internal and public search queries. Although Demon Internet was acquired by Vodafone in 2005, the crawler continued to be referenced in legacy robots.txt files and user-agent lists until around 2010. Its primary purpose was to populate a vertical search index for UK-focused web pages, not for AI training or large-scale analytics.

🌐 Technical Behavior

According to archived documentation from the Demon Internet website (demon.net), the crawler operated with a request frequency of roughly one request per 5–10 seconds per domain, respecting standard crawl delays. It used HTTP/1.1 with persistent connections and a small default crawl budget of approximately 500 pages per site per day. The IP ranges were drawn from Demon’s owned netblocks, notably 158.152.0.0/16 and 193.128.0.0/16, verified via historical BGP records. Crawl patterns followed a breadth-first strategy, starting from a seed list of UK-registered domains and following internal links up to depth 7. The bot did not parse JavaScript or AJAX content, only static HTML and CSS.

📋 robots.txt Compliance

Evidence from the Wayback Machine and the robots.txt standard documentation shows that Demon fully honored Disallow directives, as it was designed for ethical indexing. Official guides from Demon Internet instructed webmasters to block it using User-agent: Demon or User-agent: demon in robots.txt, and the crawler would exit after observing a valid disallow. No evidence exists of the bot ignoring crawl rules or bypassing restrictions.

🔍 Detection Indicators

The most common User-Agent string reported was Demon (compatible; MSIE 5.0; Windows NT; DemonWebCrawler; +http://www.demon.net) or the simpler Mozilla/5.0 (compatible; Demon/1.0; +http://www.demon.net). Behavioral fingerprints include a lack of Accept-Language headers and a fixed request pattern of exactly 5 links per second when no crawl delay is set. Identifying HTTP headers included From: [email protected] in older versions, as documented in the Demon Internet webmaster FAQ (archived at web.archive.org).

📊 Data Usage

Demon collected page titles, meta descriptions, body text, and links solely for the purpose of populating the Demon Search engine index, which returned results to users of demon.net. The data was never sold, used for AI training, or repurposed for advertising, as confirmed in the company’s 2004 privacy policy (available via the UK Web Archive). After the Vodafone acquisition, the search index was gradually deprecated, and the crawler was retired by 2011.

⚙️ Rate Limiting Policy

Today, Demon is effectively defunct, but any residual legacy crawler activity would be rate-limited aggressively because it consumes server resources for an obsolete search engine. Since no active maintenance exists, it poses a higher risk of inefficient crawl patterns, justifying threshold-based blocking after 100 requests per minute to protect modern infrastructure.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.