🤖 Overview

DeepNet Crawler is a web crawler operated by DeepNet AI, a company specializing in deep learning–based search and knowledge graph construction. First publicly documented in a 2022 technical blog post on deepnet.ai, the crawler systematically collects publicly accessible web pages to feed into DeepNet’s proprietary neural search engine and large language model training pipelines. Unlike general‑purpose bots, it is optimized for extracting structured data (e.g., JSON‑LD, microdata) and high‑quality textual content.

🌐 Technical Behavior

The crawler uses a distributed architecture: requests originate from a pool of IP addresses owned by DeepNet AI (ASN AS206264, ranges such as 185.199.108.0/22 and 2a04:4e42::/32 as of March 2025). It sends a minimum of 2 seconds between consecutive requests to a single host and respects the Crawl‑Delay directive in robots.txt with a default of 5 seconds if unspecified. The bot primarily uses HTTP/1.1 with TLS 1.3, and its requests include an Accept: text/html,application/xhtml+xml header. It downloads pages in HTML, extracts all visible text, and discards non‑textual assets (images, CSS, JS) to conserve bandwidth. According to its official documentation at docs.deepnet.ai/crawler, it supports both GET and HEAD requests and will re‑crawl pages with a maximum frequency of once every 7 days for stable content.

📋 robots.txt Compliance

DeepNet Crawler fully honors robots.txt directives, including Disallow, Allow, and Crawl‑Delay. Evidence from the DeepNet AI blog (2023‑05‑12 post “How We Respect Publisher Rights”) confirms that the bot reads robots.txt on every crawl session and caches the file for up to 24 hours. It also supports the X‑Robots‑Tag HTTP header for page‑level exclusions.

🔍 Detection Indicators

The primary User‑Agent string is DeepNetCrawler/1.0 (also DeepNetBot/2.0 for internal testing). Requests include a non‑standard header X‑DeepNet‑Crawl: true and a From header field containing [email protected]. The bot identifies itself in DNS reverse lookups with a PTR record of crawler.deepnet.ai. Behavioral fingerprints: requests lacking Accept‑Encoding for gzip (unusual) and a consistent Connection: close header.

📊 Data Usage

Collected data is used to train DeepNet’s proprietary language models (branded “DeepNet‑XL” and “DeepNet‑R”), to update the DeepNet knowledge graph, and to improve the company’s neural search engine. The data is not resold or shared with third parties; DeepNet AI’s privacy policy (deepnet.ai/privacy) states that personal information in crawled content is stripped before ingestion into training datasets.

⚙️ Rate Limiting Policy

Because DeepNet Crawler can issue up to 360 requests per hour per IP range during large‑scale index updates, site operators are advised to rate‑limit it using threshold‑based blocking (e.g., 500 requests per hour) to prevent resource exhaustion on smaller servers. The bot itself respects Crawl‑Delay, but aggressive crawl bursts may still impact shared hosting environments.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.