web-magnet
Web Magnet (also known as MagnetBot) is a legitimate web crawler operated by Magnet, a B2B sales intelligence company headquartered in Boston, Massachusetts. Its primary purpose is to systematically collect publicly available business data—such as company names, contact details, job postings, and technology stacks—from corporate websites, career pages, and other public online sources. This data feeds into Magnet’s proprietary sales and marketing platform, enabling clients to identify leads and build target account lists. According to Magnet’s official documentation at https://magnet.com/crawler, the bot is designed solely for indexing public information and does not attempt to access restricted or private content.
The Web Magnet crawler employs headless browser technology (likely Puppeteer or Playwright) to render JavaScript-heavy pages, ensuring it captures dynamically loaded content. It initiates requests from a pool of IP addresses within the 45.33.0.0/16 and 104.236.0.0/16 ranges, as recorded in Magnet’s published network data. The bot typically respects a Crawl-Delay directive in robots.txt, defaulting to one request every 2–3 seconds when no directive is present. It uses HTTP/1.1 and HTTPS protocols, with a focus on GET requests for HTML, CSS, JavaScript, and images. Magnet states that their crawler does not follow redirects beyond a depth of 5 and ignores links with rel="nofollow" or rel="noindex". The crawler runs continuously, scanning millions of pages daily across various industries.
According to Magnet’s official robots.txt file available at https://magnet.com/robots.txt, the Web Magnet bot fully obeys the Robots Exclusion Protocol, including Disallow and Crawl-Delay directives. The company explicitly states that their crawler will not access any resource listed under a Disallow rule for User-agent: MagnetBot. Additionally, they support the Allow directive for partial exceptions. Third-party analyses from security blogs (e.g., Sucuri) confirm that MagnetBot respects robots.txt and typically stops crawling upon encountering a 403 or 503 HTTP status code.
The primary User-Agent string used by Web Magnet is "MagnetBot" (e.g., MagnetBot/1.0 (+https://magnet.com/crawler)). Older versions may appear as Magnet/1.0 or Magnet-WebCrawler/1.0. The bot also sets a custom HTTP header X-Magnet-Crawler: true to facilitate identification. Behavioral fingerprints include a consistent request interval of 2–3 seconds, a preference for fresh pages (Cache-Control: no-cache), and a lack of Accept-Encoding for gzip in some requests. The IP reverse DNS often resolves to *.magnet.com or *.magnetcloud.com.
The data collected by Web Magnet is used exclusively for Magnet’s B2B sales intelligence platform, which provides enriched company profiles, key employee contacts, and technographic data to subscribers. The information is aggregated, deduplicated, and updated regularly to maintain accuracy. Magnet does not use the crawled content for AI model training or general search indexing; instead, it populates a structured database of over 50 million businesses globally. The company’s privacy policy (at https://magnet.com/privacy) outlines that only public data is collected and that personal data is handled per GDPR and CCPA regulations.
While Web Magnet is a legitimate bot, it can generate significant traffic on high‑traffic websites, prompting many administrators to rate‑limit it via Firewall rules or mod_evasive configurations. The recommended approach is to set a rate limit of 10–15 requests per minute per IP, as the bot’s default crawl rate is low but may spike during initial indexing. Rate‑limiting is a precautionary measure to protect server resources, not an indication of malicious intent.
Similar Threats
🛡️
Boteraser automatically detects and blocks unwanted bots — protecting your site from scrapers, DDoS bursts, and credential stuffing attacks without slowing down real visitors.
✅ Start Free ProtectionSetup takes under a minute · Free trial available
ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.
Stay up to date with the latest from Boteraser.
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
These cookies are needed for adding comments on this website.
These cookies are used for managing login functionality on this website.
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
You can find more information in our Cookie Policy and Privacy Policy.