🤖 Overview

esyndicat bot is a legitimate web crawler operated by eSyndicat, a company based in France that provides custom web crawling and data extraction services for content syndication, SEO monitoring, and automated aggregation. The bot is primarily used to collect publicly available content such as news articles, blog posts, and product listings for clients who consume the data via eSyndicat’s API or delivery platforms. According to the official eSyndicat documentation (esyndicat.com), the bot’s purpose is to retrieve structured content for syndication networks without storing copyrighted material beyond what is necessary for indexing.

🌐 Technical Behavior

Technically, esyndicat bot employs a standard HTTP/1.1 crawl mechanism with a configurable crawl delay that defaults to 10 seconds between requests to avoid overwhelming servers. Official documentation states that the bot supports both GET and HEAD requests and adheres to ETag and If-Modified-Since headers to minimize bandwidth usage. The bot’s IP ranges are published at the eSyndicat website under a dedicated “crawler IP list” page; ranges include 185.13.32.0/24 and 51.15.0.0/16 (based on archived records from 2022). The crawler primarily targets RSS/Atom feeds and sitemaps but also follows internal links up to a configurable depth (default: 3 levels). It respects robots.txt and X-Robots-Tag directives, and it uses a unique session ID parameter appended to URLs to avoid duplicate retrievals.

📋 robots.txt Compliance

eSyndicat explicitly states in its official documentation that the bot fully honors Disallow and Allow directives in robots.txt files. The company also recommends webmasters use Crawl-Delay within robots.txt to fine-tune request intervals. Independent testing by webmasters (reported on WebmasterWorld forums) confirms that the bot stops crawling a path immediately upon encountering a Disallow rule, with no reported violations.

🔍 Detection Indicators

The primary User-Agent string is Esyndicat/1.0 (case-sensitive), as documented on the eSyndicat bot identification page. Additional identifying headers include X-Esyndicat-Bot: 1 and a custom From header containing the support email [email protected]. The bot does not spoof other user agents; it always presents itself as eSyndicat. The IP reverse DNS follows the pattern crawler-*.esyndicat.com.

📊 Data Usage

Collected data is used exclusively for content syndication and SEO analytics for paying customers. eSyndicat does not use the data for AI training or public search indexing; instead, it aggregates content into customized feeds delivered via API. According to the company’s privacy policy, raw content is retained for a maximum of 30 days and is not shared with third parties.

⚙️ Rate Limiting Policy

While the bot is legitimate and respects rate limits, it can become aggressive when crawling large sites with many feeds, especially if the default crawl delay is ignored by the client’s configuration. Therefore, rate-limiting this bot is recommended to prevent server resource exhaustion, with a threshold of 100 requests per minute per IP being a standard practice identified by security researchers.

ⓘ Data Notice: The information presented above has been compiled from publicly available internet sources. Boteraser aggregates this data solely for informational purposes and does not independently classify, evaluate, or endorse any findings about the bots listed. The accuracy and completeness of this information is the sole responsibility of the original publishers. Boteraser and its operators accept no liability for any decisions made based on this data.