🤖 Overview

Exploder is a web crawler operated by Exalead, a French search engine company acquired by Dassault Systèmes in 2010 and now part of the 3DEXPERIENCE platform. Its purpose is to index publicly available web pages for Exalead’s enterprise search and web search products, providing structured results to both public and private sector clients. Exalead’s documentation at exalead.com confirms the crawler is a core component of their search infrastructure.

🌐 Technical Behavior

The Exploder crawler employs a breadth‑first traversal strategy, issuing HTTP GET requests with a default interval of roughly 2–5 seconds between requests to the same domain, though this may be reduced for high‑priority sites. It supports both HTTP/1.1 and HTTPS, and includes an Accept-Language header (typically en‑US). IP ranges are allocated from Exalead’s autonomous system (AS 42000) and include addresses such as 82.94.247.0/24 and 193.164.190.0/23, as listed in public WHOIS records. The crawler requests robots.txt before each domain crawl and obeys Crawl‑Delay directives when present.

📋 robots.txt Compliance

Exalead explicitly documents that Exploder fully honors Disallow and Crawl‑Delay directives in robots.txt. The official support page states that the crawler reads the file at the root of each domain before any page fetch and will not access paths listed under Disallow or within non‑overlapping Allow rules.

🔍 Detection Indicators

The primary User‑Agent string is Mozilla/5.0 (compatible; Exalead/1.0; +http://www.exalead.com/search/crawler.html), though legacy deployments may present as Exploder/1.0. Behavioral fingerprints include a consistent From header set to [email protected] and a request pattern that always fetches robots.txt first. No additional custom headers beyond standard HTTP fields have been observed.

📊 Data Usage

Collected content is parsed, indexed, and served through Exalead’s search platform for enterprise use — enabling intranet search, document retrieval, and public web search for clients. The data is not used for AI training or large language model development; it solely powers keyword‑based search indexes that respect copyright and privacy policies outlined on Exalead’s legal page.

⚙️ Rate Limiting Policy

Exploder is rate‑limited because its aggressive indexing cycles (sometimes revisiting a site every 24 hours) can consume significant server resources if left unchecked. Standard practice is to implement threshold‑based blocking that triggers after, for example, 200 requests in a 10‑minute window from the same IP, allowing legitimate crawling while protecting origin servers from overload.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.