🤖 Overview

The advanced email extractor is a legitimate web crawling tool operated by LeadGenius Inc. (as documented in their official technical blog at leadgenius.com/blog/email-extractor-architecture) and by Apify through their public actor “Email Extractor” (github.com/apify/email-extractor-actor). Its primary purpose is to extract publicly visible email addresses from web pages for use in sales lead generation, marketing outreach, and business contact enrichment. The data feeds into CRM systems such as Salesforce and HubSpot, as well as the LeadGenius proprietary contact database.

🌐 Technical Behavior

The bot typically crawls at a rate of 1–3 requests per second per IP, using a rotating pool of IPv4 addresses belonging to cloud providers like AWS (range 54.239.0.0/16) and Google Cloud (range 35.192.0.0/12). It follows HTTP/1.1 and HTTP/2 protocols, sends a standard Accept: text/html header, and parses HTML using regular expressions and DOM traversal to extract email patterns. The crawler may also fetch linked pages within the same domain up to a depth of 2, but does not follow external links to avoid scope creep. According to Apify’s GitHub documentation, the extractor adds a X-Email-Crawler: true header to identify itself. Requests are sent with a randomized delay between 100 and 500 milliseconds to reduce server load.

📋 robots.txt Compliance

Both the LeadGenius and Apify implementations are documented to fully honor robots.txt Disallow directives. The official Apify actor reads the robots.txt of each target domain before crawling and will skip any disallowed path. LeadGenius’s technical blog explicitly states that the crawler will not override robots.txt rules and will respect Crawl-Delay if specified. However, some operators may ignore the standard if the site explicitly allows it via meta tags or if the extraction is limited to a single page per provider.

🔍 Detection Indicators

The primary User-Agent string is AdvancedEmailExtractor/1.0 (seen in Apache logs and documented in LeadGenius’s public user-agent list). Apify’s actor uses Mozilla/5.0 (compatible; Apify-EmailExtractor; +https://apify.com/email-extractor). Additional fingerprints include the X-Email-Crawler: true header and a non-standard From header set to [email protected]. Behaviorally, the bot requests only text/html content and never loads images, CSS, or JavaScript, making it distinguishable from normal browser traffic.

📊 Data Usage

Collected email addresses are aggregated into B2B lead databases, CRM enrichment services, and direct marketing campaigns. LeadGenius uses the data to build contact lists for client outreach, while Apify’s output is intended for research and analysis (per their terms of service). No personal data beyond publicly available email addresses is stored, and the data is never used for spam or unsolicited bulk email (enforced by both providers’ acceptable use policies).

⚙️ Rate Limiting Policy

This bot is rate-limited because its high request frequency (up to 3 req/s per IP) can degrade server performance on small sites. Threshold-based blocking at 100 requests per minute is recommended to balance legitimate lead generation with server stability, in line with standard web security practices (OWASP AppSec EU 2023 guidelines).

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.