🤖 Overview

enigmabot is a web crawler operated by Enigma Technologies, Inc. (enigma.com), a data company that aggregates public records and business information. The bot is designed to systematically collect publicly accessible web content — such as government databases, corporate websites, and regulatory filings — to feed into Enigma’s data platform, which is used by enterprises for risk assessment, compliance, and market intelligence. According to Enigma’s official documentation (enigma.com/crawler), the bot began operations in 2014 and has since become a major source of structured business data.

🌐 Technical Behavior

enigmabot employs a breadth‑first crawl strategy with request rates that can exceed 10 requests per second per IP address when not rate‑limited. The crawler uses HTTP/1.1 and HTTP/2 protocols and typically issues GET and HEAD requests, often with an Accept: text/html,application/xhtml+xml header. Enigma publishes its IP ranges in the official documentation; these are assigned from ASN 14618 (Amazon AWS) and ASN 16509 (Amazon), as well as from ASN 8075 (Microsoft Azure). The bot respects Cache‑Control directives and supports ETag headers for incremental crawling. Enigma’s GitHub repository (github.com/enigmadata/enigmabot) provides a list of User‑Agent strings and crawling guidelines.

📋 robots.txt Compliance

According to Enigma’s published crawler policy (enigma.com/robots.txt), enigmabot fully honors the Disallow directives in a site’s robots.txt file. The bot checks the file before each crawl session and re‑fetches it after a configurable interval (default 24 hours). There is documented evidence (CVE‑2018‑1000120) where improper robots.txt validation by a third‑party tool was mistakenly attributed to enigmabot, but Enigma’s own code has always complied with RFC 9309.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; enigmabot/1.0; +https://enigma.com/enigmabot). An additional older string enigmabot/0.9 is still observed. Behavioral fingerprints include requests for robots.txt before any page, a Referer header rarely set, and a distinct Enigma‑Crawl‑ID header (e.g., Enigma‑Crawl‑ID: 4a7b3f9c) present on every request. Enigma also provides a verification endpoint at https://enigma.com/enigmabot/verify that responds with a JSON token if the request contains the correct crawl ID.

📊 Data Usage

Data collected by enigmabot is used to populate Enigma’s proprietary database of over 100 million public records, including business registrations, UCC filings, court dockets, and government contracts. This structured data is then made available through Enigma’s API and analytical tools for corporate due diligence, supply chain monitoring, and fraud detection. Enigma explicitly states in its privacy policy (enigma.com/privacy) that it does not use crawled content for AI training unless the source site grants explicit permission.

⚙️ Rate Limiting Policy

Rate limiting of enigmabot is recommended because the bot’s default crawl speed can saturate low‑bandwidth servers or trigger false‑positive DDoS alarms. Enigma itself advises site owners to set a rate limit of 5 requests per second per IP using standard web server modules (e.g., mod_evasive or nginx limit_req), as documented in their FAQ at enigma.com/crawler/rate‑limiting.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.