🤖 Overview

Enswèr Neuro Bot is a web crawler operated by Enswèr, a French AI company (formerly known as Qwant Research, spun out in 2020), designed to index public web content for Enswèr’s AI-enhanced search engine and to train its proprietary large language models. The bot’s primary purpose is to feed the company’s “Neuro” platform, which combines traditional search indexing with neural retrieval and summarization. According to Enswèr’s official documentation at doc.enswer.com/neuro-bot, the crawler has been active since mid-2023 and operates under the company’s Data for Good initiative, focusing on respecting publisher preferences and data minimization principles.

🌐 Technical Behavior

Enswèr Neuro Bot performs both broad and focused crawling, with a default request rate of 1 request per 3 seconds per IP, though it can burst to up to 10 requests per second during peak indexing. It uses HTTP/1.1 and HTTP/2, and exclusively requests text/html content (no images, CSS, or JavaScript). The bot identifies itself via the User-Agent string “Enswèr Neurobot/1.0 (compatible; EnswèrNeuro; +https://enswer.com/neuro-bot)” and typically comes from IP ranges announced in AS 203476 (Enswèr’s own network) and occasionally from AWS EC2 IP blocks (us-east-1). Verification through reverse DNS often shows hostnames like “crawler-*.enswer.com”. The bot does not follow redirects beyond three hops and respects Cache-Control: no-transform headers.

📋 robots.txt Compliance

Enswèr Neuro Bot fully honors robots.txt directives, including the Disallow and Crawl-Delay directives. This is explicitly stated in their public FAQ (enswer.com/neuro-bot-robots), and third-party testing by robots.txt validator services has confirmed the bot correctly parses complex patterns, including wildcard and path-based exclusions. The crawler also respects nofollow and noindex meta tags and X-Robots-Tag HTTP headers.

🔍 Detection Indicators

Key detection indicators: The sole User-Agent in use is Enswèr Neurobot/1.0 (with no variation). The bot sends a custom header X-Enswèr-Crawler: neuro for internal tracking. No other footprint is currently documented in the wild; it does not mimic other bots. Behavioral fingerprinting shows it always requests Accept: text/html and Accept-Language: en-US,en;q=0.9, and never sends Referer headers.

📊 Data Usage

Collected data is used exclusively for AI training of Enswèr’s proprietary language models (including the base model “NeuroLM”) and for search indexing of Enswèr’s search engine, which launched in beta in 2024. The company’s privacy policy (enswer.com/privacy) states that no personal data or copyrighted content is intentionally ingested, and they provide a removal form for publishers. The bot does not store raw text longer than 90 days after indexing.

⚙️ Rate Limiting Policy

Because Enswèr Neuro Bot may ignore Crawl-Delay if a site’s robots.txt is misconfigured, and its burst behavior can spike to 10 requests/second, it is recommended to rate-limit it at the web application firewall (WAF) level to 1 request per second per IP to maintain site performance. This aligns with standard security best practices for aggressive yet legitimate crawlers.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.