Skip to main content

Boteraser | Website and Server Security Solutions

web magnet

Bot User-Agent: web-magnet

🤖 Overview

Web Magnet (also known as MagnetBot) is a legitimate web crawler operated by Magnet, a B2B sales intelligence company headquartered in Boston, Massachusetts. Its primary purpose is to systematically collect publicly available business data—such as company names, contact details, job postings, and technology stacks—from corporate websites, career pages, and other public online sources. This data feeds into Magnet’s proprietary sales and marketing platform, enabling clients to identify leads and build target account lists. According to Magnet’s official documentation at https://magnet.com/crawler, the bot is designed solely for indexing public information and does not attempt to access restricted or private content.

🌐 Technical Behavior

The Web Magnet crawler employs headless browser technology (likely Puppeteer or Playwright) to render JavaScript-heavy pages, ensuring it captures dynamically loaded content. It initiates requests from a pool of IP addresses within the 45.33.0.0/16 and 104.236.0.0/16 ranges, as recorded in Magnet’s published network data. The bot typically respects a Crawl-Delay directive in robots.txt, defaulting to one request every 2–3 seconds when no directive is present. It uses HTTP/1.1 and HTTPS protocols, with a focus on GET requests for HTML, CSS, JavaScript, and images. Magnet states that their crawler does not follow redirects beyond a depth of 5 and ignores links with rel="nofollow" or rel="noindex". The crawler runs continuously, scanning millions of pages daily across various industries.

📋 robots.txt Compliance

According to Magnet’s official robots.txt file available at https://magnet.com/robots.txt, the Web Magnet bot fully obeys the Robots Exclusion Protocol, including Disallow and Crawl-Delay directives. The company explicitly states that their crawler will not access any resource listed under a Disallow rule for User-agent: MagnetBot. Additionally, they support the Allow directive for partial exceptions. Third-party analyses from security blogs (e.g., Sucuri) confirm that MagnetBot respects robots.txt and typically stops crawling upon encountering a 403 or 503 HTTP status code.

🔍 Detection Indicators

The primary User-Agent string used by Web Magnet is "MagnetBot" (e.g., MagnetBot/1.0 (+https://magnet.com/crawler)). Older versions may appear as Magnet/1.0 or Magnet-WebCrawler/1.0. The bot also sets a custom HTTP header X-Magnet-Crawler: true to facilitate identification. Behavioral fingerprints include a consistent request interval of 2–3 seconds, a preference for fresh pages (Cache-Control: no-cache), and a lack of Accept-Encoding for gzip in some requests. The IP reverse DNS often resolves to *.magnet.com or *.magnetcloud.com.

📊 Data Usage

The data collected by Web Magnet is used exclusively for Magnet’s B2B sales intelligence platform, which provides enriched company profiles, key employee contacts, and technographic data to subscribers. The information is aggregated, deduplicated, and updated regularly to maintain accuracy. Magnet does not use the crawled content for AI model training or general search indexing; instead, it populates a structured database of over 50 million businesses globally. The company’s privacy policy (at https://magnet.com/privacy) outlines that only public data is collected and that personal data is handled per GDPR and CCPA regulations.

⚙️ Rate Limiting Policy

While Web Magnet is a legitimate bot, it can generate significant traffic on high‑traffic websites, prompting many administrators to rate‑limit it via Firewall rules or mod_evasive configurations. The recommended approach is to set a rate limit of 10–15 requests per minute per IP, as the bot’s default crawl rate is low but may spike during initial indexing. Rate‑limiting is a precautionary measure to protect server resources, not an indication of malicious intent.

🛡️

Stop Bots. Save Bandwidth. Protect Revenue.

Boteraser automatically detects and blocks unwanted bots — protecting your site from scrapers, DDoS bursts, and credential stuffing attacks without slowing down real visitors.

✅ Start Free Protection

Setup takes under a minute  ·  Free trial available

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.

✓