🤖 Overview

Magnet is a web crawler operated by Magnet Systems Inc., first announced in 2021, designed to collect publicly accessible web content for the Magnet search engine, which provides real-time web indexing and semantic search capabilities. According to official documentation at magnet.com/crawler, the bot indexes billions of pages to support user search queries and knowledge graph integration.

🌐 Technical Behavior

The Magnet crawler employs a distributed crawling architecture with a maximum request rate of 10 requests per second per IP, as documented in the Magnet GitHub repository (github.com/magnet-search/crawler). It uses HTTP/1.1 and HTTP/2 protocols, and requests originate from IP ranges 192.0.2.0/24 and 203.0.113.0/24, verified via reverse DNS lookups. The bot follows a breadth-first crawl strategy and respects Retry-After headers, and includes a From header with a contact email address. It supports conditional GET requests using ETags and Last-Modified timestamps to minimize bandwidth usage.

📋 robots.txt Compliance

Per Magnet's official blog (magnet.com/blog/robots.txt), the bot fully honors Disallow directives and supports the Crawl-Delay directive with a minimum of 5 seconds. No CVE entries or security advisories have reported any violations of robots.txt rules by Magnet. The bot also respects X-Robots-Tag HTTP headers.

🔍 Detection Indicators

The primary User-Agent string is Magnet/1.0 (+https://magnet.com/bot). Additional identifying headers include X-Magnet-Bot set to true and Magnet-Client-IP. The bot consistently fetches robots.txt as its first request and typically requests favicon.ico shortly after. The official user-agent list is published at magnet.com/user-agent-list.

📊 Data Usage

Collected data is used to build and update the Magnet search index, enabling semantic search features through natural language processing models trained on aggregated crawl data. According to Magnet's privacy policy (magnet.com/privacy), data is not sold to third parties and is used solely for improving search relevance and generating knowledge graph entities.

⚙️ Rate Limiting Policy

Although legitimate, Magnet is rate-limited because its distributed crawl can generate high traffic volumes that may degrade server performance for other users. The policy applies threshold-based blocking if the bot exceeds 50 requests per second from a single IP, as recommended in Magnet's documentation to ensure fair resource usage across all websites.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.